http://www.realclimate.org/index.php/archives/2009/12/please-show-us-your-code/

[...] calculations done by Scafetta & West (in the journals Geophysical Research Letters, Journal of Geophysical Research, and Physics Today) have inspired the idea that the recent warming is due to changes in the sun, rather than greenhouse gases. [...] Gavin and I published a paper in Journal of Geophysical Research, where we tested the general approach used by Scafetta & West, and tried to repeat their analysis. We were up-front about our lack of success in a 100% replication of their work, but we argue that the any pronounced effect - as claimed by Scafetta & West - should be detectable even if the set-up is not 100% identical.

However, Scafetta does not accept our analysis and has criticized me for lacking knowledge about wavelet analysis - he tells me to read the text books. So I asked him to post his code openly on the Internet so that others could repeat our test with their code. That should settle our controversy.

Looks like the denialists have something to hide...

The venerable BBC is reporting that a survey of light emitted from white dwarfs showed that between 1% and 3% had material (such as silicon) falling into the star on a continuous basis, potential evidence of dead worlds and asteroids. On this basis, the authors of the study speculate that the same percentage of mainstream stars in the active part of their life will have rocky matter. This is not firm evidence of actual planetary formation, as asteroids would produce the same results, but it does give an upper bound and some idea of what a lower bound might be for planetary formation.

Aside from being a useful value for Drake's Equation, the rate of planetary formation would be valuable in understanding how solar systems develop and what sort of preconditions are required for an accretion disk of suitable material to form.

Because the test only looked for elements too heavy to have been formed in the star, we can rule out the observations being that of cometary debris.

Four fireballs, glowing blue and orange, were visible last night over the skies of the Carolinas on the southeast coast of the United States, followed by the sound of an explosion described as being like thunder. Reports of hearing the noise were coming in from as far afield as Connecticut. There is currently no word from NASA or the USAF as to what it could be, but it seems improbable that anything non-nuclear the military could put up could be heard over that kind of distance. It therefore seems likely to be a very big meteorite.

The next question would be what type of meteorite. This is not an idle question. The one slamming into the Sudan recently was (a) extremely big at an estimated 80 tonnes, and (b) from the extremely rare F-class of asteroid. If this new meteorite is also from an F-class asteroid, then it is likely associated with the one that hit Sudan. This is important as it means we might want to be looking very closely for other fragments yet to hit.

The colours are interesting and allow us to limit what the composition could have been and therefore where it came from. We can deduce this because anything slamming through the atmosphere is basically undergoing a giant version of your basic chemistry "flame test" for substance identification. We simply need to look up what metals produce blue, and in so doing we see that cadmium does produce a blue/violet colour, with copper producing more of a blue/green.

Other metals also produce a blue glow and tables of these colours abound, but some are more likely in meteoric material than others. Cadmium exists in meteorites. Well, all elements do, if you find enough meteorites. but it exists in sufficient quantity that it could produce this sort of effect. (As noted in the chemmaster link, low concentrations can't be detected by this method, however this is going to be vastly worsened by the fact that this isn't a bunsen burner being used and the distance over which you're observing is extreme.)

Ok, what else do we know? The fireballs were also orange. Urelites, such as the Sudan impact, contain a great deal of calcium, which burns brick-red, not orange. This suggests we can rule out the same source, which in turn means we probably don't have to worry about being strafed the way Jupiter was with the Shoemaker-Levy comet (21 impacts).

What can we say about it, though? Well, provided the surviving fragments didn't fall into the ocean, it means every meteorite hunter on the planet will be scouring newspaper stories that might indicate where impacts occurred. Meteoric material is valuable and anything on a scale big enough to be heard across the entire east coast of the US is going to be worth looking for. It had split into four in the upper atmosphere, so you're probably looking at a few thousand fragments reaching ground level that would exceed a year's average pay.

Someone on TECHWR-L posted a link to this paper (under the paradoxical title "The Cupertino Effect"), which is about how Excel's autocorrect feature can corrupt statistical analysis of genetic data if/when Excel "makes the wrong assumption" about an entry based on how it looks:

When processing microarray data sets, we recently noticed that some gene names were being changed inadvertently to non-gene names. A little detective work traced the problem to default date format conversions and floating-point format conversions in the very useful Excel program package. The date conversions affect at least 30 gene names; the floating-point conversions affect at least 2,000 if Riken identifiers are included. These conversions are irreversible; the original gene names cannot be recovered.

As the author points out, this can cause gene names to come back in analyses as "unknown," because "[a] default date conversion feature in Excel ... was altering gene names that it considered to look like dates. For example, the tumor suppressor DEC1 ... was being converted to '1-DEC.'"

The authors also note that there is a problem with "RIKEN [4] clone identifiers of the form nnnnnnnEnn" being converted to a floating-point number.

The paper also gives some idea of the devastating scale of the problem and its significance for people doing these sorts of analyses: "A non-expert user might well fail to notice that approximately 3% of the identifiers on a microarray with tens of thousands of genes had been converted to an incorrect form, yet the potential for 2,000 identifiers to be transmogrified without notice is a considerable concern."

As far as I know personally and can glean from the paper, the autocorrect and/or conversion feature is nearly impossible to disable completely, and can only be worked around, possibly unsuccessfully 100% of the time. This suggests that perhaps Excel is not the tool of choice for doing these sorts of analyses. (Does the spreadsheet application in OpenOffice work differently?)

Heise Security reports that A team of researchers from Bonn University and RWTH Aachen University have analysed the notorious Storm Worm botnet, and concluded it certainly isn't as invulnerable as it once seemed. Quite the reverse, for in theory it can be rapidly eliminated using software developed and at least partially disclosed by Georg Wicherski, Tillmann Werner, Felix Leder and Mark Schlösser. However it seems in practice the elimination process would fall foul of the law.

Meta-moding, I got to a comment talking about Comaq in present tense - puzzled, I looked at the date:

by jeffwah on 15.01.1999 20:42
Attached to: Compaq to bundle Linux and provide support

Yup, close to a decade old. The others comments are somewhat younger (like 2004).

In light of the NIST complaint that there are so many applicants for their cryptographic hash challenge that a good evaluation cannot be given, I am curious as to whether they have adequately defined the challenge in the first place. If the criteria are too loose, then of course they will get entries that are unsuitable. However, the number of hashes entered do not seem to be significantly more than the number of encryption modes entered in the encryption mode challenge. If this is impossible for them to evaluate well, then maybe that was also, in which case maybe we should take their recommendations over encryption modes with a pinch of salt. If, however, they are confident in the security and performance of their encryption mode selections, what is their real objection in the hashing challenge case?

But another question one must ask is why there are so many applicants for this, when NESSIE (the European version of this challenge) managed just one? Has the mathematics become suddenly easier? Was this challenge better-promoted? (In which case, why did Slashdot only mention it on the day it closed?) Were the Europeans' criteria that much tougher to meet? If so, why did NIST loosen the requirements so much that they were overwhelmed?

These questions, and others, look doomed to not be seriously answered. However, we can take a stab at the criteria and evaluation problem. A strong cryptographic hash must have certain mathematical properties. For example, the distance between any two distinct inputs must be unconnected to the distance between the corresponding outputs. Otherwise, knowing the output for a known input and the output for an unknown input will tell you something about the unknown input, which you don't want. If you have a large enough number of inputs and plot the distance of inputs in relation to the distance in outputs, you should get a completely random scatter-plot. Also, if you take a large enough number of inputs at fixed intervals, the distance between the corresponding outputs should be a uniform distribution. Since you can't reasonably test 2^512 inputs, you can only apply statistical tests on a reasonable subset and see if the probability that you have the expected patterns is within your desired limits. These two tests can be done automatically. Any hash that exhibits a skew that could expose information can then be rejected equally automatically.

This is a trivial example. There will be other tests that can also be applied automatically that can weed out the more obviously flawed hashing algorithms. But this raises an important question. If you can filter out the more problematic entries automatically, why does NIST have a problem with the number of entries per-se? They might legitimately have a problem with the number of GOOD entries, but even then all they need to do is have multiple levels of acceptance and an additional round or two. eg: At the end of human analysis round 2, NIST might qualify all hashes that are successful at that level as "sensitive-grade" with respect to FIPS compliance, so that people can actually start using them, then have a round 3 which produces a pool of 3-4 hashes that are "classified-grade" and a final round to produce the "definitive SHA-3". By adding more rounds, it takes longer, but by producing lower-grade certifications, the extra time needed to perform a thorough cryptanalysis isn't going to impede those who actually use such functions.

(Yes, it means vendors will need to support more functions. Cry me a river. At the current scale of ICs, you can put one hell of a lot of hash functions onto one chip, and have one hell of a lot of instances of each. Software implementations are just as flexible, with many libraries supporting a huge range. Yes, validating will be more expensive, but it won't take any longer if the implementations are orthogonal, as they won't interact. If you can prove that, then one function or a hundred will take about the same time to validate to accepted standards. If the implementations are correctly designed and documented, then proving the design against the theory and then the implementation against the design should be relatively cheap. It's crappy programming styles that make validation expensive, and if you make crappy programming too expensive for commercial vendors, I can't see there being any problems for anyone other than cheap-minded PHBs - and they deserve to have problems.)

Found this interesting site, which is focussing on developing grid computing systems for gaming. The software they seem to be using is a mix of closed and open source.

This could be an important break for Linux, as most of the open source software being written is Linux compatible, and gaming has been the biggest problem area. The ability to play very high-end games - MMORGs, distributed simulators, wide-area FPS, and so on, could transform Linux in the gaming market from being seen as a throwback to the 1980s (as unfair as that is) to being considered world-class.

(Windows machines don't play nearly so nicely with grid computing, so it follows that it will take longer for Microsoft and Microsoft-allied vendors to catch up to the potential. That is time Linux enthusiasts can use to get a head-start and to set the pace.)

The question that interests me is - will they? Will Linux coders use this opportunity of big University research teams and big vendor interest to leapfrog the existing markets completely and go straight for the market after? Or will this be seen as not worth the time, the same way that a lot of potentially exciting projects have petered out (eg: Open Library, Berlin/Fresco, KGI, OpenMOSIX)?

Two hundred and sixty seven tapes of previously unheard electronic music by Delia Derbyshire have been found and are being cataloged.

For those unfamiliar with Delia Derbyshire, she was one of the top pioneers of electronic music in the 1950s and 1960s. One of her best-known pieces was the original theme tune to Doctor Who. According to Wikipedia, "much of the Doctor Who theme was constructed by recording the individual notes from electronic sources one by one onto magnetic tape, cutting the tape with a razor blade to get individual notes on little pieces of tape a few centimetres long and sticking all the pieces of tape back together one by one to make up the tune".

Included in the finds was a piece of dance music recorded in the mid 60s, examined by contemporary artists, revealed that it would be considered better-quality mainstream today. Another piece was incidental music for a production of Hamlet.

The majority of her music mixed wholly electronic sounds, from a sophisticated set of tone generators and modulators, and electronically-altered natural sounds, such as could be made from gourds, lampshades and voices.

Someone has trawled through YouTube and flagged not only the episodes of The Tripods, but also all fan productions, fan cine footage and fan photography of the series. How so, can't you buy it on DVD? Only the first season, the second exists only in pirated form at scifi conventions, and of course the fan material doesn't exist elsewhere at all. The third season, of course, was never made, as the BBC had a frothing xenophobic hatred of science fiction at the time. (So why they made a dalek their general director at about that time, I will never know...)

What makes this exceptionally annoying is that the vast bulk of British scifi has been destroyed by the companies that produced it, the vast bulk of the remainder has never seen the light of day since broadcast, and the vast bulk of what has been released has been either tampered with or damaged in some other way, often (it turns out later) very deliberately, sometimes (again it turns out later) for the purpose of distressing the potential audience.

I've nothing against companies enforcing their rights, but when those companies are acting in a cruel and vindictive fashion towards the audience (such as John Nathan Turner's FUD of audiences being too stupid to know what they like, or too braindead to remember what they have liked), and the audiences vote with their feet, on what possible grounds can it be considered justified for those companies to (a) chain the audience to the ground, and (b) then use the immobility of the audience to rationalize and excuse the abuse by claiming the audience isn't going anywhere?

I put it to the Slashdot Court of Human/Cyborg Rights that scifi fans are entitled to a better, saner, civilized explanation, and that whilst two wrongs can never make a right, one wrong is never better.

Well, it now appears the University of Manchester in England has built 1nm transistors on graphene. The article is short on details, but it appears to be a ring of carbon atoms surrounding a quantum dot, where the quantum dot is not used for quantum computing or quantum states but rather for regulating the electrical properties. This is still a long way from building a practical IC using graphene. It is, however, a critical step forward. The article mentions other bizare behaviours of graphene but does not go into much detail. This is the smallest transistor produced to date.

This has been the dullest April 1st Slashdot has ever had. Has its funny bone been surgically removed? For those who miss the days of OMG!!! PONIES!!!, I think I've found the only site that comes close in being very very strange.....

Gary Gygax, July 27, 1938 - March 4, 2008

This is an odd one for me -- depending on how you look at it, I was in the RPG world for many years, but not of it, or of it, but not in it. Most of my friends are gamers, some of them quite hardcore for years. When I was a teenager, I wrote a paper RPG. (In retrospect, it seems as though what I was reaching for -- completely independently of anything, since I didn't know the people involved and they didn't know me, and there was a lapse of several years involved -- was what someone else eventually wrote as Vampire: The Masquerade, although what I came up with was closer to GURPS and, admittedly, ultimately unplayable. At least I got class credit for it.) I had a front-page entry in the Daily Illuminator (under the name of the person whose e-mail I was borrowing at the time, since I didn't have one of my own -- scroll down to October 2). I even owned a set of dice. (I've since given them away.)

However... Aside from a few abortive attempts at LARPing in the mid-90s, I've never actually played a pen-and-paper RPG.

However again, if it hadn't been for the RPG culture, in which I was at least an interested bystander if not an out-and-out participant, I never would have become the (geeky-but-at-home-with-it) person I am today.

So I owe one to Gary Gygax. Thanks, man. I've never even played your game, but you still changed my life.

One of the most fascinating things I've observed in searching for Open Source projects available for whatver I'm doing at the time is the huge disparity of what is available, how it is used and who is interested.

An obvious place to start is in the field of electronics. Computer-based tools are already used to build such stuff, so it's a natural replacement, right? Well, almost. There are tools for handling VHDL, Verilog and SystemC. There are frameworks for simulating both clock-based and asychronous circuits. You can do SPICE simulations, draw circuit diagrams, download existing circuits as starting points or places of inspiration, simulate waveforms, determine coverage and design PCBs. OpenCores provides a lot of fascinating already-generated systems, SUN provides the staggering T1 and T2 UltraSPARC cores, and the Sirocco 64-bit SPARC. This field has probably not got anywhere near what it needs, but it has a lot.

Maths is another obvious area. Plenty of Open Source tools for graphing, higher order logic, theorum provers, linear algebra, eigenvalues, eigenvectors, signal processing, multiple-precision, numerical methods, solvers for all kinds of other specific problem types, etc.

What about astronomy? That requires massive table data crunching, correlation of variations, moving telescopes around with absolute precision - things computers tend to be very good at. There are a few. Programs for capturing images are probably the most common, although some telescopes provide software for controlling telescopes, obtaining data and performing basic operations. Mind you, how much more than this does one need in software? Some things are better done in hardware (for now, at least) because the software hasn't the speed. Yes, the control software seems a little specialized, but it'd be hard to make something like that general-purpose.

Chemistry. Hmmm. Lots of trivial stuff, more educational than valuable - periodic tables, 3D models of molecules, LaTeX formatting aids. There's a fair amount on the study of crystals and crystallography, which is as much chemistry as it is physics, but there's not a lot else. Chemistry involves a lot of tables (which would be ideal for a standardized database), a lot of mathematical equations, formulae, graphing, measuring and correlating all sorts of data, the consequences of different filtering and separation techniques, the wavelength and intensity of energies, analysis of the results of atomic mass spectrometry or other noisy data, etc. I see the underlying tools for doing some (but not all) of these things, but I don't see the heavy lifting.

Archaeology has very few non-trivial tools. Some signal processing for ground-penetrating RADAR, but there are virtually no tools out there that could be useful for helping with interpretation. In fact, most RADAR programs don't interpret either but display the result on a small LCD screen. Nor do any tools exist for correlating interpretations (other than manually via an extremely naive - for this purpose - GIS database). There's a few scraps here and there, but signal analysis and GIS seem to be about it, and those were mostly developed for mining companies and tend to show it.

Biology has plenty of DNA sequencing code. By now, Slashdotter should be able so sequence eith own DNA, not pay someone a thousand to do it. You mean, those aren't enough, that you need more hardware? And a lot more software? It's an important step, but it's not unique.

Mechanical Engineering. I haven't seen anything of any significance.

Geology. Not really, beyond the same software for Archaeology, but using it for find seams in rock.

Psychology: Nada.

Psychiatry: None.

Sports: Lots of software getting used, but little of it is open source.

Result - those who gain with the least to lose and the most to win make the change. Those who feel like there's no benefit from changing what they're doing will continue doing what they're doing. My suggestion? There are gaping holes in Open Source. Fill them in.

This is an interesting (to me) piece of work that I've been asked to do. Using open-source software to analyze data from both ground-penetrating radar and magnetometers, open-source GIS software for tracking archaeological finds, open-source modeling software to produce archaeologically and technically sound reconstructions, and then use a mix of open-source virtual reality software and open-source web technology to provide both the raw and the visually interpreted data in a form that is of practical use to experts and non-experts alike.

If that sounds like a complex task, it is. The site is extremely convoluted, there is a wealth of data that is currently in a highly unusable form, and what is meaningful to an expert is not necessarily the least bit useful or usable to a non-expert (and vice versa). Currently, there is a lot of skepticism by The Powers That Be that such a project would even be possible. My first task, then, is to produce an example. My impossible mission is to convert the few scraps of information published on medieval aisled halls, along with the very limited archaeological finds from the site in question, into the dual format of raw information and virtual reality.

On the one hand, the limited information means that the first part is relatively easy. An online archaeological GIS-enabled database may not be trivial, but all the software needed can - at least - be found on Freshmeat and the amount of data entry is relatively small. The second part is tougher. Again, open-source VR software does exist, but it is one thing to enter known values that can be verified into a database, it is entirely another to derive values that are implied and logically required but for which there is no direct evidence at all.

There is a catch. Virtual reality is great for producing models you can walk through, but it's generally pretty lousy at telling you if said model violates the laws of physics. Given that I can hardly build my own medieval aisled hall, I know of no other method besides hand-cranking through the numbers for validating the predicted structure. Suggestions would be extremely welcome, as would any idea on how I could either use the open-source approach for the hall design, or how I could use something like BOINC to automate the validation of a virtual landscape.

Technically, this is fun - I'm getting to do some reasonably original work - but original work is necessarily far more demanding in terms of research and application than run-of-the-mill work. Mind you, I only have myself to blame - the archaeologists have been satisfied so far with producing a web-based diary of major finds, plus entering the data on a completely unusable regional database. Such are the hazards of pointing out that you can do better! :)