Malacandra.me

Latest Posts

So, if she weighs the same as a duck ... [Class M]

OK. Taking on logical flaws in Wall Street Journal op-ed items is about as difficult as shooting fish in a barrel, but I can't let Matt Ridley's latest affront to common sense pass without firing off a few rounds — for practice if nothing else.

Read the rest of this post... |

Read the comments on this post...

Also check out the featured ScienceBlog of the week: Inside the Outbreaks on the ScienceBlogs Book Club


comments

Rated “euw” for wicked bug sex [bioephemera]

coverstewart100.jpg

Amy Stewart's new book Wicked Bugs: The Louse That Conquered Napoleon's Army & Other Diabolical Insects takes a fairly trivial concept - a collection of historical anecdotes and icky factoids about dangerous insects - and executes it remarkably well. The book is well-written and has a non-cloying sense of humor ("she's just not that into you," begins the section relating how female praying mantids eat the males). Briony Morrow-Cribbs lends her insect illustrations (see a NYTimes slideshow of her ink illustrations here), and Anne Winslow's design plays off the concept of a vintage textbook - more Hogwarts than Harvard, perhaps, but still cute. I love the red-and-crimson cover.

Read the rest of this post... |

Also check out the featured ScienceBlog of the week: Inside the Outbreaks on the ScienceBlogs Book Club


comments

Hasselblad’s 200 megapixel H4D-200MS camera now shipping, breaks your bank

Hasselblad's H4D-200MS is yet another upgrade to its H4D camera line that we want badly, but wouldn't have a need -- or a budget -- for. This latest piece of kit is nearly identical to the 50 megapixel H4D-50MS externally, but internally adds an astonishing 150 more megapixels to the mix -- yes, that's an astounding 200 megapixels! Don't feel forgotten just yet if you have the older model, though, as Hasselblad can upgrade your sensor for €7,000 (roughly $10,000) -- a great deal considering the full kit costs €32,000 (about $45,000). With that said, we should note that details are null on the medium-format sensor's native resolution -- similar to the multi-shot feature on the 50MS, the new 200MS combines six shots to create ridiculously detailed 200 megapixel still images, and it handles less intensive photos with a four-shot 50 megapixel still mode. Should you dare to venture outside the studio, there's a 50 megapixel single-shot live mode for quick shots as well. Excuse us while we wrap our heads around the sheer number of pixels; you'll find more info after the break in the meantime.

Continue reading Hasselblad's 200 megapixel H4D-200MS camera now shipping, breaks your bank

Hasselblad's 200 megapixel H4D-200MS camera now shipping, breaks your bank originally appeared on Engadget on Thu, 26 May 2011 06:02:00 EDT. Please see our terms for use of feeds.

Permalink   |  sourceHasselblad  | Email this | Comments
comments

How to Create a Web Spy with a PHP Crawler

Crawler, spider, bot, or whatever you want to call it, is a program that automatically gets and processes data from sites, for many uses.

Google, for example, indexes and ranks pages automatically via powerful spiders, crawlers and bots. We have also link checkers, HTML validators, automated optimizations, and web spies. Yeah, web spies. This is what we will be doing now.

Actually I don’t know if this is a common term, or if its ever been used before, but I think it perfectly describes this kind of application. The main goal here is to create a software that monitors the prices of your competitors so you can always be up to date with market changes.

You might think “Well, it is useless to me. You know, I’m a freelancer, I don’t have to deal with this ‘price comparison’ thing.” Don’t worry, you are right. But you may have customers that have a lot of competitors they want to watch closely. So you can always offer this as a “plus” service (feel free to charge for it, I’ll be glad to know that), and learn a little about this process.

So, let’s rock!

1 – Requirements

  • PHP Server with linux – We need to use crontab here, so it is better to get a good online server
  • MYSQL – We will store data with it, so you will need a database

2 – Basic crawling

We will start by trying a basic crawling function: get some data. Let’s say that I sell shoes, and Zappos is my competitor (just dreaming about it). The first product I want to monitor is a beautiful pair of  Nike Free Run+. We will use now fopen to open the page, fgets to read each line of the page and feof to check when we need to finish the reading. At this time, you need to have fopen enabled in your server (you can check it via phpinfo ). Our first piece of code will be:

<?php
	if(!$fp = fopen("http://www.zappos.com/nike-free-run-black-victory-green-anthracite-white?zlfid=111" ,"r" )) {
		return false;
	} //our fopen is right, so let's go
	$content = "";

	while(!feof($fp)) { //while it is not the last line, we will add the current line to our $content
		$content .= fgets($fp, 1024);
	}
	fclose($fp); //we are done here, don't need the main source anymore
?>

At this point, if you echo the $content you will notice that it has all page contents without any CSS or JS, because on zappos site they are all with relative paths.

Now we have the content, we need to process the product price.

How do you know the difference between price and  other ordinary data in our page? Well, it is easy to notice that all prices must have a “$” before them, so what we will do is get all data and run a Regular Expression to see which prices where we have a dollar sign,  we have on that page.

But our regular expression will match every price on the page. Since Zappos is a good friend of spies, it has made the “official” price as the first, always. The others are just used in JavaScript, so we can ignore them.

Our REGEX and price output will be something like this:

<?php
//our fopen, fgets here

//our magic regex here
	preg_match_all("/([$][0-9]*[,]*[.][0-9]{2})/", $content, $prices, PREG_SET_ORDER);
	echo $prices[0][0]."<br />";
?>

Wow, we now have the price. Don’t forget the other prices, we will need them if Zappos changes something in their site.

3 – Save data in MYSQL

Let’s prepare our DB to receive this data. Let’s create a table called zappos. Inside of it we will have four columns:

  • ID - Primary key on this table
  • Date - When data was stored. It’s good to store this so you can do some reports.
  • Value - Value that you’ve found
  • Other_Values - Values that aren’t what you want, but it’s important to store them so if the site owner changes the code you have a “backup” of the possible values

In my phpmyadmin I’ve created a database called spy, and inside it my table zappos, this way:

CREATE TABLE IF NOT EXISTS `zappos` (
  `ID` int(5) NOT NULL AUTO_INCREMENT,
  `Date` date NOT NULL,
  `Value` float NOT NULL,
  `Other_Values` char(100) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  PRIMARY KEY (`ID`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ;

Once you’ve created your table, we will start adding some data. So we will need to do a mysql connect in our PHP and prepare our prices to be saved.

Since all our data is not perfect floats, we need to prepare it so we will have just numbers and a dot.
To connect in our db we will use mysql_connect, and after we will use mysql_select_db to select “spy” and then we can do our mysql_query to save or get our data.

<?php

//preparing to save all other prices that isn't our "official" price
	$otherValues = "";
	foreach ($prices as $price) {
		$otherValues .= str_replace( array("$", ",", " "), '', $price[0]); //we need to save it as "float" value, without string stuff like spaces, commas and anything else you have just remove here
		$otherValues .= ","; //so we can separate each value via explode when we need
	}

//if someday Zappos changes his order (or you change the site you want to spy), just change here
	$mainPrice = str_replace( array("$", ",", " "), '', $prices[0][0]);

//lets save our date in format YYYY-MM-DD
	$date = date('Y\-m\-d');

	$dbhost  = 'localhost';
	$dbuser  = 'root';
	$dbpass  = '';
	$dbname  = "spy";
	$dbtable = "zappos";

	$conn = mysql_connect($dbhost, $dbuser, $dbpass)
		or die ('Error connecting to mysql');
		echo "<br />Connected to MySQL<br />";

		$selected = mysql_select_db($dbname)
			or die( mysql_error() );
			echo "Connected to Database<br />";

			//save data
			$insert = mysql_query("
						INSERT INTO `$dbname`.`$dbtable` (
							`ID` ,
							`Date` ,
							`Value` ,
							`Other_values`
						)
						VALUES (
							NULL , '$date', '$mainPrice', '$otherValues'
						);
					");
			//get data
			$results = mysql_query("SELECT * FROM $dbtable");

	mysql_close($conn);

//all data comes as MYSQL resources, so we need to prepare it to be shown
	while($row = mysql_fetch_array($results, MYSQL_ASSOC)) {
		echo "ID :{$row['ID']} " .
			 "Date : {$row['Date']} " .
			 "Value : {$row['Value']}";
		echo "<br />";
	}

?>

4 – Smarter spy with Crontab

Well, with crontab we can schedule some tasks in our (linux) system so it runs automatically. It is useful for backup routines, site optimizing routines and many more things that you just don’t want to do manually.

Since our crawler needs some fresh data, we will create a cron job that runs every day at 1am. On net.tuts+ we have a really good tutorial on how to schedule tasks with cron, so if you aren’t too familiar with it, feel free to check it out.

In short, we have command lines that we could use for it, (second is my favorite):

#here we load php and get the physical address of the file
#0 2 * * * says that it should run in minute zero, hour two, any day of month, any month and any day of week
0 2 * * * /usr/bin/php /www/virtual/username/cron.php > /dev/null 2>&1

#my favorite, with wget the page is processed as it were loaded in a common browser
0 2 * * * wget http://whereismycronjob/cron.php

5 – Let’s do some pretty charts

If you are planning to use this data, just a db record won’t be too useful. So after all this work we need to present it in a sexier way.

Almost all our jobs here will be done by the gvChart jQuery plugin. It gets all our data from tables and make some cool charts out of it. What we have to do actually is print our results as a table, so it can be used by gvChart. Our code this time will be (download our demo for more info!):

<?php
	$dbhost  = 'localhost';
	$dbuser  = 'root';
	$dbpass  = '';
	$dbname  = "spy";
	$dbtable = "zappos";

	$conn = mysql_connect($dbhost, $dbuser, $dbpass)
		or die ('Error connecting to mysql');

		$selected = mysql_select_db($dbname)
			or die( mysql_error() );

			//get data
			$results = mysql_query("SELECT * FROM $dbtable ORDER BY `ID` DESC LIMIT 15");

			mysql_close($conn);

			$dates  = array();
			$values = array();
			while($row = mysql_fetch_array($results, MYSQL_ASSOC)) {
				$dates[] = "{$row['Date']}";
				$values[] = "{$row['Value']}";
			}

			echo "<table id='real'>";
				echo "<caption>Real Prices on Zappos.com</caption>";
				echo "<thead>";
					echo "<tr>";
						echo "<th></th>";
						foreach($dates as $date) {
							$date = explode('-', $date);
							echo "<th>" . $date[2] . "</th>";
						}
					echo "</tr>";
				echo "</thead>";
				echo "<tbody>";
					echo "<tr>";
						echo "<th>" . $date[0] . "-" . $date[1] . "</th>";
						foreach($values as $value) {
							echo "<td>" . $value . "</td>";
						}
					echo "</tr>";
				echo "</tbody>";
?>

Are you hungry yet?

I think there’s a lot to improve on yet. You could, for example, do a “waiting list” of urls so you could crawl a lot of URL’s with a single call (of course each URL could have his own REGEX and “official price”, if they are from different sites).

And what do you think we could improve?

comments

Chinese prisoners forced to produce virtual gold, real profits for their guards

The virtual goods economy of massively multiplayer online games may be thriving, but it's also stimulating an undesirable side-effect: exploitation. A former detainee at a prison in Heilongjiang province, China, has told the Guardian about how he was habitually forced into playing MMOs like World of Warcraft for the collection of loot, which the prison guards would then resell online for as much as ¥6,000 ($924) per day. Such totals would be the product of up to 300 inmates working 12-hour daily shifts, though predictably they saw none of the profits themselves. The unnamed source was at a "re-education through labor" camp where the usual toil would involve actual, rather than virtual, mining. The profitability of the online market has seemingly inspired prison bosses to move with the times, however, with business being so brisk that the computers "were never turned off." A Chinese government edict from 2009 is supposed to have introduced a requirement that online currencies only be traded by licensed entities, but it's believed that the practice of using prisoners in this fashion continues unabated.

Chinese prisoners forced to produce virtual gold, real profits for their guards originally appeared on Engadget on Thu, 26 May 2011 05:14:00 EDT. Please see our terms for use of feeds.

Permalink   |  sourceGuardian  | Email this | Comments
comments

Naturopathy invades the heartland [Respectful Insolence]

Many are the times I've referred to homeopathy as The One Quackery To Rule Them All. Because homeopathic remedies diluted greater than 12 C (12 serial hundred-fold dilutions) have been diluted more than Avagadro's number, they are incredibly unlikely to have even a single molecule of starting compound in them. That makes them water. Given that the vast majority of homeopathic remedies are, in fact, only water, they are the perfect quackery, and any effects due to homeopathy are nonspecific and placebo effects. More recently, I've pointed out that, because you can't have naturopathy without homeopathy, naturopathy is the grab bag quackery for which no quackery is too quacky to be included in the grab bag. I, of course, respond to homeopaths and naturopaths, "Get into the f-ing sack."

Personally, I like to use acceptance of homeopathy as a litmus test for credulity, knowing that anyone who believes in homeopathy--even understanding what it is--has a serious credulity problem. Naturopathy has a serious credulity problem. Even so, naturopathy seems to be making inroads in a lot of places, being the "respectable" form of quackery. Heck, its practitioners even give themselves the title of "doctor," and the "ND" after their names is really close to the "MD" after the names of real doctors. Maybe that's why naturopathy is among the most successful alt-med "disciplines" at insinuating its way into academic medical centers, "integrating" itself with science-based medicine. Of course, I never could figure out how "integrating" quackery with science-based medicine does anything other than taint SBM with pseudoscience.

Be that as it may, the other day I saw more evidence of just how prevalent the infiltration of pseudoscience has become. Normally, naturopathy and other forms of "alternative medicine" are associated with large population centers full of people who, because they live in large population centers yearn for the "natural" and earnestly believe that, just because it's natural it must be better. Now, it's infiltrating places like rural Montana:

Read the rest of this post... |

Read the comments on this post...

Also check out the featured ScienceBlog of the week: Inside the Outbreaks on the ScienceBlogs Book Club


comments

Tagedit jQuery Plugin with Auto Complete List

Tagedit jQuery Plugin by Oliver Albrecht offers an easy way to add, edit and delete lists of keywords. Keywords can be brought to the user with a autocomplete list. I really like the clean and sleek design of the tags. This work is licensed under a Creative Commons Attribution 3.0 Unported License.

tagedit

Requirements: jQuery Framework
Demo: http://tagedit.webwork-albrecht.de/
License: Creative Commons License

Sponsors

Using What Font is you can identify the font you are looking for!

Aactis Shopping Cart: easy, fast and reliable. Check for special offers.

comments

“Ask the wizard”: Grandpa Simpson babbles nonsense again

I think that many people believed that George W. Bush and Sarah Palin were members of a new breed of conservative politicians who were mean as snakes, dumb as posts and spoke gibberish instead of English. Not true. They've been around forever:

"The AARP, I mean, come come," [Alan] Simpson said to an audience of Washington insiders. "If you can't understand that when I was a freshman at the University of Wyoming, there were 17 people paying into the system and one taking out, and today there are three people paying into the system and one taking out -- if you can't understand that it was never set up as a retirement program, it was an income supplement which became a retirement, if you can't understand it was never structured to handle disability insurance, it couldn't exist with that burden on it. If you can't understand it didn't take care of kids at 22 going to college, we can't make it."

Simpson went on to reference a recent interview with Huffington Post reporter Ryan Grim, who presented Simpson with evidence that one of the statistics he deployed in his Social Security arguments was misleading.

"Now the great sharpshooters are out there and the cat food commission cats and all those guys using these distorted figures," Simpson told the crowd. "And I always say, look, if you torture statistics long enough, eventually they'll confess."

In truth, Social Security was indeed established as a retirement program.

Lately, Simpson has been fond of claiming that the average life expectancy when Social Security was created was just 63 years of age, much lower than today. But the figure that actually matters for Social Security finances is the life expectancy for people who live to 65, the age at which benefits kick in. That number hasn't changed much since 1940.

In an interview with The Huffington Post following his remarks, Simpson reiterated his attack.

"I was talking about the guy who called me and went through this exercise of sharpshooting," Simpson told HuffPost. "And if he can't understand a couple or three things then there's no help. Forget all the crap he's going through and know that if you -- if 17 people were paying into this system in 1950 and one taking out, today there are three paying in and one taking out. I'd like you to refute that."

Can't someone draw him a picture or something? Or just direct him to the social security administration web site? Here's the explanation of the life expectancy question:

If we look at life expectancy statistics from the 1930s we might come to the conclusion that the Social Security program was designed in such a way that people would work for many years paying in taxes, but would not live long enough to collect benefits. Life expectancy at birth in 1930 was indeed only 58 for men and 62 for women, and the retirement age was 65. But life expectancy at birth in the early decades of the 20th century was low due mainly to high infant mortality, and someone who died as a child would never have worked and paid into Social Security. A more appropriate measure is probably life expectancy after attainment of adulthood.

As Table 1 shows, the majority of Americans who made it to adulthood could expect to live to 65, and those who did live to 65 could look forward to collecting benefits for many years into the future.

Or maybe he could read Nancy Altman's book The Battle For Social Security where she spells out the worker to retiree issue so plainly that even Alan Simpson should be able to wrap his addled mind around it:

… all pension programs that require a period of employment for eligibility, private as well as public, show similar ratios at the start, because all newly covered workers are paying in, but no one in the newly covered group has yet qualified for benefits. The president could just as accurately have said that in 1945, the ratio of works to beneficiaries as 42 workers paying in for every one beneficiary or the equally accurate but misleading ratio from 1937, 26 million workers paying in for about a dozen beneficiaries.

… what is important is not the worker-to-beneficiary ratio at the start of the program but the ratio when the program reaches maturity. Consistent with the meaninglessness of the 16-to-1 factoid, the worker-to-beneficiary ratio was halved to eight workers for every beneficiary within five years, and by 1975, the ratio was where it is today. The 1994-1996 advisory council had not agreed on much, but it made one very valuable contribution. Its report included the appendix that stated that "the fundamental ratio of beneficiaries to workers was fully taken into account in the 1983 financing provisions, and, as a matter of fact, was known and taken into account well before that...

That's right. And as a matter of fact some of the people who created the program are still around to testify about what they intended. Also from Altman's book:

Bob Myers watched Bush on television from his home right outside Washington and stared in disbelief. In 1934, not only could Myers foresee the world as it changed, he had forecast these changes with great specificity. He was the one who had crunched the numbers for Roosevelt's Social Security proposal. Myers and Otto Richter, the senior actuary with whom he had worked, had been extremely farsighted. Myers knew, in 1934, that people in the twenty-first century would live longer and draw benefits longer.

As it turned out, Myers and Richer were a shade too conservative in their projections, believing the percentage of the population that would be elderly in the future would actually be higher than it turned out to be. Specifically, in 1934, he and Richter projected that, in year 2000, 12.7 percent of the population would be age 65 or older. How accurate were they? According to the 2000 census figures, the percentage of those aged 65 and over was 12.4 percent of the population.

So no, these Social Security actuaries over the years have not been total morons who need Alan Simpson to point out that they were total dopes by not foreseeing that people would probably live longer in the future or that there would be more people paying into the system than collecting from it in the early years of the program's existence.

And by the way, the unexpected post-war baby boom was dealt with by having all of us pre-pay a ton of money into the system that would make up for the fact that we were a larger than normal demographic. We invested that money in an unusual financial instrument called a US Treasury bill, the same one that Bill Gates and Lloyd Blankfein and The People's Republic of China buy because they are considered the world's safest investment. It's not like we didn't all do the prudent thing.

Huffington Post concludes with this:

"Repeating a false claim over and over again does not make it true," said Frank Clemente of the Strengthen Social Security Campaign, a coalition over 270 national and state organizations dedicated to protecting Social Security from benefit cuts. "Those who continue to use this canard show they are more interested in tearing down Social Security rather than making it stronger. Social Security has a huge surplus today but a long-range gap in 25 years that can be closed relatively painlessly if the richest two percent of Americans started paying Social Security taxes on all their wages -- like nearly all other Americans do."

Good idea. But they've all threatened to go on strike because it just wouldn't be worth it to them to work anymore and our whole civilization would come down around ears.

Finally, please, someone tell Simpson to pipe down and take his nitro glycerin before he has a heart attack from all the gibberish:

"I know all the stuff [Ryan Grim] goes through. Its like gymnastics! Yes and we've done distributional analysis. Ask him if he knows what that is! Ask the wizard if he knows what distributional analysis is! We did that. And then ask him what we did for the seniors, for the older old and the people who are in poverty. Ask the wizard all that and then get back to me."

He then shouted, "I'm through!" and walked away.

God, if only ...


.

comments

Food Storage and Preservation Class Starting Next Week! [Casaubon’s Book]

It seemed up here in the north that spring would never come - and now we're headed rapidly into that time of year when everything is ripe and abundant in our gardens and at local farms, and learning to put food up can make it possible for you to enjoy summer in winter, and continue eating locally as long as possible. It can be overwhelming when you start preserving, so if you'd like a friendly voice to walk you through it, please join us, starting next Tuesday!

The class is on-line and asynchronous, and you can participate at your own pace. Every week we'll have projects involving what's growing in gardens and markets to get you familiar with the basics of preserving the harvest, and also help you build up food security by building up a reserve of stored food.

My hope is that at the end of the class, everyone will have a plan for how they want to go about increasing their food storage reserves, and will have tried the major methods of food storage. You will be able to watch the jars increase as the class goes on, and you'll be ready for peak preserving season.

Here's a rough syllabus:

Week 1, May 31 - Introduction to Food Storage, How much, where to put it, and how? Can I afford this? Low energy overview of food preservation methods. Storing Water, making space.

Week 2, June 7: Water bath canning 101, Preserving with Salt, Sugar and Honey, Bulk purchasing, sourcing local foods, finding food to preserve, what food storage can and can't do.

Week 3, June 14: Dehydration basics, Tools you need and where to get them, Menu making and how to get people to eat from your pantry, Setting up your kitchen for food storage, Storing herbs and spices, Sourdoughs and grain ferments, Preserving foraged foods.

Week 4 June 21: Lactofermentation; Special needs and health issues; Storing food for children, the elderly, pregnant and lactating women; Storing medications, gluten-free storage; Basic dairy preservation and cheesemaking; Building up your pantry and Managing your reserves.

Week 5, June 28: Pressure Canning; Beverages, Teas and Drinks; Preserving in Alcohol, Coops and Community Food Security; More Menus and Recipes; Root Cellaring and in-Garden Storage.

Week 6, July 5: Season extension, Preserving Meats, Sprouting, The next Steps, Getting Your Community Involved, Teaching others, Food Preservation as a Cottage Industry.

We will try and track the seasonal produce coming in, support each other as we experiment with new techniques and build up our pantries as we go - and have a lot of fun! If you are interested in joining, cost of the class is $150 or equivalent barter if we can come to a mutually agreeable arrangement.. I also have three scholarship spots remaining for low income participants who would otherwise be unable to afford to take the class. If you'd like to donate to the scholarship fund, just let me know - 100% of your donation goes to making classes available to low income participants. Email me to enroll or with questions at jewishfarmer@gmail.com.

Ok, off to do something with the rhubarb coming in!

Sharon

Read the comments on this post...

Also check out the featured ScienceBlog of the week: Inside the Outbreaks on the ScienceBlogs Book Club


comments

GOP Congressional Members

Caucus behind closed doors with Paul Ryan to respectfully discuss the tricky position that his Bold and Serious Plan to Destroy Medicare puts them in.

comments

‹ First  < 3132 3133 3134 3135 3136 >  Last ›