When verifying a large file's checksum, I usually ...
Displaying poll results.18623 total votes.
Most Votes
- What's the highest dollar price will Bitcoin reach in 2024? Posted on February 28th, 2024 | 8481 votes
- Will ByteDance be forced to divest TikTok Posted on March 20th, 2024 | 7898 votes
Most Comments
- What's the highest dollar price will Bitcoin reach in 2024? Posted on March 20th, 2024 | 68 comments
- Will ByteDance be forced to divest TikTok Posted on March 20th, 2024 | 20 comments
IDK (Score:1)
I dunno but so far it seems like Cowboy Neal has been doing a good enough job.
Re:IDK (Score:5, Informative)
Run it with administrative privileges without delay.
Re: IDK (Score:2)
Linux Mint is not amused.
Re: (Score:2)
Really, though, the question wasn't "do you verify" or in "what (if any) circumstances should you verify" large files, it is "When verifying, I usually ....".
Re: IDK (Score:2)
That would be "sha1sum", not "shasum" (Score:2)
I guess this poll was not tested....
Re: (Score:1)
Why are you such a bigot? sha512sum has feelings too!
Re: (Score:2)
That one is actually a Perl-wrapper, not a checksum application itself.
I check everything (Score:5, Funny)
BOFH
Re: (Score:2)
Re: (Score:2)
Don't forget to grind up the platters and dissolve the dust in acid! Always practice safe computing, kids!
PGP signature (Score:5, Interesting)
I'd prefer just a PGP signature for the file. Next would be a PGP signed file containing MD5 or SHA values, but that seems redundant to me when you could just sign the original file.
Unauthenticated checksums are useless for establishing trust.
Re: (Score:2)
Unauthenticated checksums are useless for establishing trust.
I don't use md5sum for trust as much as making sure I actually downloaded the entire file. I generally only do it for very large files like ISOs, if I suspect a problem, or if it doesn't run the first time. Most of the time if it's some form of archive or install package, it will fail to run if it's incomplete.
Re: (Score:1)
Sometimes you can verify the checksum from a different source/mirror, which does provide for at least a little trust.
The point of signing the checksums instead of the file is that it's faster to verify, provides the same security/integrity guarantees, and you can have signed checksums for lots of different files and only one signature, instead of having a signature for each file.
Also, I don't see how it could be "redundant". Redundant would be doing both.
Re: (Score:2)
'Also, I don't see how it could be "redundant". Redundant would be doing both.'
A PGP signature is essentially a hash of the file signed by a private key.
Distrubuting a file containing a hash and then PGP signing that file is hashing the file, then hashing the hash of the file, then signing the second hash. Redundant.
As to the speed difference, I just ran a 4GB ISO file through sha256sum which took 25 seconds. Using gpg to generate a detached signature the same file took 42 seconds. That's a DVD sized file.
Re: (Score:2)
Good explanation. I agree with everything except this part:
Then they have to VISUALLY (and prone to error) compare 2 long and complicated numbers.
They could also just direct the output to a couple of temporary files and then diff them.
Why go to all that trouble? It's a one-liner:
[ `sha256sum some.iso|cut -d " " -f1` == `head -4 some.iso.sha256|tail -1|cut -d " " -f1` ] && echo 'Checksums match'
Enjoy.
Re: (Score:2)
Yes, that's much more convenient and intuitive than:
gpg -v some.iso.asc
Re: (Score:2)
Does that work if all you're provided is an ISO and a SHA256 file?
Re: (Score:2)
Nope that only works if you are provided an ISO and a detached PGP signature. That's why I'm saying supplying a detached signature of the original file is my preferred method (if the original file needs to remain usable without PGP).
Re: (Score:2)
Just checking.
Re: (Score:2)
From a security point of view, there is no point in providing an file and its hash on the same source.
That's why you usually have the sha256 provided on a website (not in a file) with a link to the file on a mirror/CDN (a less trustable source, but more convenient for large files).
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
I usually invest some energy finding a copy of their key from a source not directly on the same site/servers but still under their control. In the end though, you're right that a PGP key can't be trusted unless your web of trust validates it.
Re: (Score:3)
What even totally zero-web-of-trust PGP gives you over "here's your .tgz, here's the SHA1 from the same server" is reproducibility. I can't 100% acertain that key 0xDEADBEEF really is the Apache Tomcat project signing key unless I know someone who knows someone. But, I *could* verify that the Tomcat tarballs I downloaded three and six months ago and the one I downloaded today were in fact signed by someone with same key 0xDEADBEEF, so if their site has been hacked it's a very long-running hack.
Re: (Score:2)
For MS ISOs, I use two sources. I first use the checksums they post, and I put the MD5 and SHA values into the usual search engines, and usually something pops up if the ISO is correct. It isn't 100%, but at least it means an attacker would have to astroturf, showing that their bogus ISO is legit, as well as hack Microsoft's site to change the checksum values.
Re: (Score:2)
I like having both, if possible. The .sig/.asc files are good for the initial vetting of all downloads, while the signed manifest is good to ensure that once everything is downloaded, bitrot hasn't gotten any files when stashed on the NAS, and one has long since lost the public signing key, and doesn't have time to fish it out. Plus, the signed manifest file can be sent to itconsult.co.uk's time stamper as an additional security check for date/time. This way, an attacker not just has to bag the signing r
Most files no point (Score:3, Interesting)
Most of my big downloads are video, if it runs then fine. If it's an installer they tend to come with a digital signature - which would obviously fail if it's been corrupted/tampered with post-signing, if it's an archive it'll fail during extraction. Pretty much the only time I might do it is when downloading an ISO, but only because it's a plain sector dump with no integrity check.
Re: (Score:1)
Furthermore, every file that you download through HTTPS or Torrents will have its file integrity checked by software, so if you got it, it's fine. No need to manually check checksums.
Re: (Score:2)
Re: (Score:2)
That's not a fault in the protocol.
That's a fault in the tool implementation.
Re: (Score:2)
TCP itself has integrity checking. But while these protocols are effective at making sure the file is transferred correctly, it doesn't guarantee that the file wasn't corrupted at the source or destination.
Torrents are usually fine because the checksums are in the torrent file, which is separate. However, HTTPS won't help you much against accidental corruption compared to any other TCP-based protocol. What HTTPS protects you against is eavesdropping and malicious tempering during transfer..
Re: (Score:2)
yeah. that all works in principle.
I've had corruption from copying a file without any errors being evident. The copy was over the local network, but that wasn't the culprit: it was the SATA controller on the destination drive (technically, one SATA controller involved in the raid).
But, hey, who needs to bother with validating results? If it doesn't occur to you how a problem could crop up then it simply can't happen, amiright?
Whatever they provide (Score:5, Interesting)
I would prefer PGP or SHA512/256. But most of the time the only posted hash is MD5, and even that's somewhat rare. Typically I just have to trust that the listed filesize on the site is correct, and see if that matches.
I must be getting my big files from the wrong places, since people here seem to be under the impression that hashes and checksums are regularly provided, but I rarely see that
Missing option: Just the file size? (Score:2)
Actually, I normally check the file size first, and then I look at the file with an appropriate application, from the beginning, middle, and end of the file to see if it's okay. It would have to be a pretty important file to justify the effort of running and confirming checksums.
who checks the checksum? (Score:2)
But how do you know the checksum is correct? I hope there's a checksum available so I can verify that the checksum of the checksum checks out before I go through all that effort of checking if the checksum matches the file.
Re: (Score:2)
Re: (Score:1)
Can you name any file-transport protocol with such built-in checksum?
Both UDP and TCP contain data payload checksums. One could argue that this eliminates the need for higher-level checksums/hashes on e.g. FTP or HTTP traffic, depending on your needs (this won't establish data trust, only verify the message content).
Re: (Score:2)
Well, not every time. This made for a fun few weeks at work a while back: https://tech.vijayp.ca/linux-k... [vijayp.ca]
It's checksums all the way down (Score:5, Funny)
Re:It's checksums all the way down (Score:5, Funny)
How much check could a checksum sum if a checksum could check sum?
Have you been eating at the Restaurant At the End of the Universe?
Re: (Score:2)
Re: (Score:1)
Clearly it could check some.
Re: (Score:1)
None of the above (Score:2)
Sigh. Like most polls, the either-or choices in this one are stupid.
Both MD5SUM and SHA1SUM are obsolete and brain-dead stupid to use. The only checksum that makes sense is SHA512SUM, and that is what I use.
No skill or experience here (Score:2)
Doesn't anyone just eyeball it, old fasioned style? Maybe pull out a loupe just to be sure?
Re: (Score:2)
I used to burn it on a CD and read it back from there using my naked eye, but not any more since I haven't had a CD burner for the better part of a decade...
Indeed skills like eyeballing the music on a CD is truly a lost skill!
PowerShell (Score:5, Informative)
Re: (Score:2)
What the hell to you think you're doing?? We never talk about useful features of PowerShell on this site! Ever! Or any other useful features coming from Redmond. Only bugs deemed features by their marketing department. We only BASH BASH BASH
Re: (Score:1)
I agree. It was a good concept to be able to pick fields by name.
But the implementtion is terrible.
And once I discovered that output is refomatted depending on destination it was the nail in the coffin.
Re: (Score:2)
once I discovered that output is refomatted depending on destination it was the nail in the coffin.
ls removes color if you pipe it. It is a sane thing to do, and I would guess you can override it in PowerShell too?
Like you I think the implmentation is pretty crap, I like the idea though maybe if they take it abit further.
Re: (Score:2)
In Powershell the output is always formatted in the last step of an operation because it is an object-oriented shell and everything is technically an object or array of objects.
Once you understand that everything is an object, the behavior of Powershell makes a lot of sense and you can use the object-oriented nature to do things that would be fairly difficult to do.
If you really want to work with p
Re: (Score:2)
Piping object is actually quite nice when you get used to it.
I really don't understand the hate ppl have towards something as mundane as tech
I recently used hashing to track down bad hardware (Score:2)
Recently my primary home computer had been having maddening, completely unpredictable stability problems. I could swap out parts, but I was lacking a way to verify that the problem went away with the swapped part. Sometimes the system would go for days without crashing, sometimes it was every 5 minutes. Sometimes it was during gaming, sometimes web browsing.
I was finally able to get a lock on the problem by generating a 100 GB file and repeatedly running md5 on it. With the bad part (RAM) I was getting
Re: (Score:1)
Excellent, a good usage and proof that hashes are not always the gospel.
Re: (Score:2)
Re: (Score:2)
To fill in some more background I had already run the Windows memory tester for several hours (over night, in fact) and gotten a false negative result. That gave me false confidence in my RAM so I was actually looking for hard disk problems when I started testing with MD5. I got the idea because Steam seemed to constantly detect that my games had corrupt files and redownload them, even after it had just finished redownloading. After being able to reliably reproduce the bad MD5 result I did some Prime95 t
Re: (Score:2)
Re: (Score:2)
When testing, you only assert there was no errors on that run. This depends on the quality of the testing scenario and on the reproducibility of the errors that may happens.
Some kind of hardware errors may happen irregularly and even Memtest may not catch them on a single run.
So if you do not catch errors, that doesn't mean there is no error. Just that you did not catch any on that run.
Re: (Score:2)
Re: (Score:2)
Use an NSA proxy server as a filter. (Score:1)
Use an NSA proxy server as filter. If they're happy with it, it must be safe.
Where's MY option! (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Parity bit? That's luxury. We would just figure out where the line noise was by looking at the file with a hex editor, and fix it from there, perhaps writing better assembly to fix the chunk of code that got munged than the original programmer had in place.
Re: (Score:2)
My option is missing too (Score:1)
I use Grøstl checksum because it has a funny name.
What is "large"? (Score:2)
What I feel you're really asking in this poll... (Score:3)
When I read through the poll question and possible answers, it felt as though it was a euphemism for something else.
Poll: When using birth control, do I...
1) Only use a condom (1180 votes / 15%)
2) Only use birth control pills (435 votes / 5%)
3) Use both a condom and birth control pills, and maybe others (409 votes / 5%)
4) Use a condom, but who cares if it's expired (1257 votes / 16%)
5) Use birth control pills, but only use them occasionally (509 votes / 6%)
6) Just do it twice, and if no one gets pregnant, score! (140 votes / 1%)
7) Real people don't use birth control; they just accept the consequences. (2686 votes / 36%)
8) Cowboy Neal prescribes my birth control! (826 votes / 11%)
It's even funnier if you consider the poll numbers were to be the same in both polls.
None of the above (Score:2)
Correct checksums do not prove that the server hasn't been compromised and the file and checksum changed.
I put the checksum to good use at https://www.virustotal.com/#se... [virustotal.com] and test against 56 AntiVirus applications.
Re: (Score:2)
Virus total also allows people to comment on a file and mark it good/bad as well. It is a quite useful tool. If you are using Windows and download the latest autoruns version, it actually will scan and check every running process against Virustotal's database (both in an online and offline mode) to help catch a rogue/unwanted process.
What about... (Score:1)
So no one checks the SHA256 sum then its signature against a known valid version of the signer's public key?
I feel all alone on my small isle.
Re: (Score:2)
Re: (Score:1)
I don't really get your comment. Of course you need to trust the signer's key which is what I mean by a "known valid version of the signer's key", and the basis for that is not another user's trust unless you really trust that user as well.
However you can look at multiple sources for the key before giving it some trust (mail archives, wayback machine, which key signed previous versions, etc). Best is to actually verify the signature in person, but that's not always possible.
Note that in any case it can't be
"As a user..." (Score:2)
...I expect all download managers, backup managers, etc. to do this by default if there's an .md5 or any other extension file "around" the source: e.g. for websites, in the page's remainder links; for any file system or FTP, something in the same folder... If there's no such expected file around, the application performing the large file operation should warn the user.
Fact of the matter is, as a user, nobody SHOULD have to care about disk/cpu/memory/transport reliability. IT engineers keep telling other peo
This poll is on the sidebar (Score:1)
Started doing this, atleast on iso's. (Score:2)
Hardware Store (Score:1)
I only install from sites using https (Score:2)
With a properly signed cert, following the chain of trust, etc, etc.
Anyone going through the trouble of a Man in the Middle attack on an insecure connection, can spoof the checksum as well. I mean seriously.
The real way to verify a download (Score:2)
Only first and last three characters. (Score:2)
I only check the first three and last three characters of an md5 hash. The odds of an accidental hash collision on a file that was corrupted during transmission is 1 in 16,777,216.
Re: (Score:2)
About security, checksum are there when the file is hosted on a less trustable host (such as a mirror) than the main host that provides you the checksum and the link.