Forgot your password?
typodupeerror
User Journal

Journal: GPG signature: first remove whitespace+non-printables 1

Journal by KWTm

I've settled on a hopefully credible way to sign my posts without needing to worry about how word-wrapping will mess up the GPG signature. Before signing my posts, I will remove spaces, tabs, newlines, and any other non-printable characters before signing.

That means that the GPG signature of my posts will be invalid as-is (since the text of the posting itself retains the whitespace and non-printables). That means you will have to remove the whitespace/non-printables yourself before running through the GPG verification to see that it is a valid signature. This is how to do it:

put the plain text into PlainTextFile, and the signature (including beginning and end lines) into SignatureFile, and use this command:
tr -cd [:graph:] <PlainTextFile | gpg --verify SignatureFile -

Don't forget the single hyphen at the end of the line, which has a space before it.

User Journal

Journal: New OpenPGP key as of 2010-01-15

Journal by KWTm

Okay, this time I'm remembering to post my new key before my old key expires. So, this replaces the key that I posted in 2008.

The idea is that, if I get kicked out of this account (or someone takes over this account), then I can set up a new Slashdot account and use this key to prove that I'm the same user.

Here it is. In case Slashdot formats it, remember that if there are any extra-long lines, those should be broken up into individual lines. Each line of this OpenPGP key is the same length and contains no spaces.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.6 (GNU/Linux)

mQGiBEtQZNgRBADGdWKqfED1ZHpS+FbcTF5yq0IpaK6YFrUOxtvY6mhNqBFG/+9M
NyZQxQK9Bfx38KBkeu8H4AHeDHnx19wfrRidS2O/6yrcwahN9i/ZAxF2VgjZafZe
PuYnBXCnIkiPcwzBXOo5MkwKHdqKv6yQYbMWBQolVZLJaxG3W7MAGrHF3wCgpXPu /Vm080e3HWIo46C4GyFe9jkEALPNDenlhnQzTC72nhgEbGdVaYZK5k11gw7kaCSZ
YQougUbBzHZX6yrdfvP7qgtudtpUGNgr4CQc56A+2iTn1eibyw/q/FjlW0IcVXez
f3aIxRMbQO+5vXXnDbED5/dZQvSPNuXzR5q25arRYbaMcoH0YzJSQLg/vkbSHwmn
pKkNA/wOBZl3hb23x56WqPaXb0xyHnY6u3/EPd9Lpn54A0vBcvSqzeXIgrESv6kz
bE7C2FMkp+hLP91brPi51gioSrFs6dDrnRw+adUBMjbApjyCy8XN3wk2IV2Oijud
ca9cFo7Ijsp7oXA4B1S8rvtjc/t1uqiC1VIUtiJJ6cCzQQ6ry7Q2S2EgV2FpIFRh
bSAoMjAxMCBKYW4gMTUgS21haWwpIDxrYXdhaS1rYkB0YW1seWxpbi5uZXQ+iGYE
ExECACYFAktQZNgCGyMFCQWjmoAGCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRAu
dz05U7/JnhBXAJ4namsGw21iZtLo0TsoYlcnmHRyWwCfZuV4cnJFDus6SjVQ0I1S
4ykXq765Ag0ES1Bk5BAIAPrxJ0ljf/g6cmGtqclwe1UBRHusFapl7wjR4TVoysaU
MB7wD8m41gSSKL4LlirlFkL+EceKr7AVxev2af5qgjV+2Px4TS5VvWcFvd1jPcMs
G8D0j9MNM+lE5SSfOwl/gqcaQoMdTsgx21zsDlgJyithiZbWG0mc9UCufdM8+MeE
BpFiykd9rUcuMSMUaYI5dHSZSv+3ZOqpYC1YI4SRYDck/t13GpcA+hElxiJ3PwWv
0PIfb9QkT3qwICgAJ+E28qjrLHqdGy2bALp/HSlLf2QbwUUj6d6DH7lRXzuV0ZqQ
MI7bnouag0b+bIEWUfK7muyEY1P7Nh5WDLmQ1Ta3mDsAAwYH/j1uwBUjpAmj0SiY
pIwVl+nN0fCyVU45gm7CasBENAJ7jG+ZalK83etZ87jhd/T/pIuKvOso1V1g3zAv
O7BVfXbm94ezE9t2tuv/k9QQoMqfajflBv8rHLzPdyzhc5RaAGbV3fP0+7Q5q01W
lpjs4Dkiau/bdg1B7XBdmb8rovB0gfvD+EhWRr+VWWAAV1NdqDKHyYXcdGfRhkjK
lxUDMc7H2l7H1VUoHXnoL1C+8/KOk9RfuKXdIsGztkJrookykOvZ8GTB91eUL2QH
yEfMetNmAG7NzJM2d0NtZyyULRqujPrUcDsJDhD2u66lDH2FElIlT+Zely5FW+aj
WsA9rMiITwQYEQIADwUCS1Bk5AIbDAUJBaOagAAKCRAudz05U7/JnoR8AKCk1m6r
InrF5Gr0XmiJvfi4hAylNwCdGRRJOtJSEogYds0dwvhrrBUv9NA=
=Kmjs
-----END PGP PUBLIC KEY BLOCK-----

User Journal

Journal: processing GnuCash files (XML): a Python script 3

Journal by KWTm

I was asked for the Python code I wrote to process GnuCash files. I figured I may as well post it publicly so that more than one person can benefit from the work I put in. I cleaned it up a bit, but if you have any questions, please contact me. My program here is called GnuCash_Fix.

As mentioned in the comments: When GnuCash imports QFX/OFX/QIF files (say, for a bank account or credit card account), each transaction is between 2 accounts but GnuCash can only identify 1 of the accounts, that of the bank account/credit card account in question. If the QFX/OFX/QIF file is imported without further intervention, the other account is marked as "Unbalanced" and is assigned to this account (for example, "Unbalanced-USD" if in US Dollars) after doing Main Menu > Actions > Check & Repair > All transactions.

This program tries to identify the other account for each transaction (e.g. "groceries" or "salary") depending on the description of the transaction and optionally the range of the amount. It also finds two complementary unbalanced transactions and matches them together. E.g.
# transaction #1 on 2003-06-21, for $123.45 from Account A to ...somewhere unknown!!!; and then
# transaction #2 on 2003-06-21, for $123.45 from somewhere unknown ... to Account B.
Gee, I wonder what could have happened with those two seemingly unrelated transactions???? (duhhh...)

Works for GnuCash v2.2.6, and also for earlier versions that I used (I can't tell which versions now, but at least one of them was a v1.x.x version).

The main tricky thing I found out from working with GnuCash files is that the tags contain the colon character (':') which Python's XML libraries don't like, so I used "sed" to convert them to double underscores ("__")

In addition to depending on some publicly available software such as "sed" and various Python libraries, it also depends on two Python modules of my own creation. I will post these in subsequent entries. It's easy to rewrite the code so it does not have these dependencies, but if you wanted to make sure the code was working as-is, you'll need these modules. One module is just a debugging script (decide whether to display debugging messages depending on a pre-set "debugging level of detail), and the other is just a personalized version of "getopt". (In fact, I think you can just replace it with the official Python "getopt" --but the version with GNU compatibility.)

Oh, it also depends on a pre-defined file containing parameters that allow this GnuCash_Fix program to identify and classify the transactions. This is called descr_acct.py, and I will post it in a subsequent entry.

#!/usr/bin/env python
#
# Copyright KWTm 2006-2009
 
# released under GPL v3. See http://www.gnu.org/copyleft/gpl.html
# Those who don't like this license may contact me for other licensing options, such as my special I-Don't-Like-GPL-And-Want-To-Give-You-Lots-Of-Money license.
 
#
# When GnuCash imports QFX/OFX/QIF files (say, for a bank account or credit card account), each transaction is between 2 accounts but GnuCash can only identify 1 of the accounts, that of the bank account/credit card account in question. If the QFX/OFX/QIF file is imported without further intervention, the other account is marked as "Unbalanced" and is assigned to this account (for example, "Unbalanced-USD" if in US Dollars) after doing Main Menu > Actions > Check & Repair > All transactions.
 
# This program tries to identify the other account for each transaction (e.g. "groceries" or "salary") depending on the description of the transaction and optionally the range of the amount.
#
# Also finds two complementary unbalanced transactions and matches them together. E.g.
# transaction #1 on 2003-06-21, for $123.45 from Account A to ...somewhere unknown!!!; and then
# transaction #2 on 2003-06-21, for $123.45 from somewhere unknown ... to Account B.
# Gee, I wonder what could have happened with those two seemingly unrelated transactions???? (duhhh...)
 
# Works for GnuCash v2.2.6, and also for earlier versions that I used (I can't tell which versions now, but at least one of them was a v1.x.x version).
 
# Dependencies:
# standard Python libraries, especially xml.dom.ext and xml.dom.minidom
# my own modules: kwdebug, kwgetopt (but this script can be easily rewritten to eliminate these dependences)
# standard GNU/Linux command-line utilities: gzip, sed
 
# The Python xml.dom.minidom module doesn't like colons as part of the XML tag name, so we need to s/:/__/ or something.
 
"""
I'll set up a naming convention:
 
_Ac = instance of Account class
_Class = a class
_Cr = instance of Criterion class
_Gn = instance of Gnucash class
_Tr = instance of Transaction class
_Ts = instance of Timestamp class
_Xn = xml_node
 
"""
 
import solitime
import sys
sys.path.append('/usr/lib/python%s/site-packages/oldxml' % sys.version[:3])
# The above 2 lines are necessary because Ubuntu 8.04 moved the python-xml package, breaking old code. Stupid.
import xml.dom.minidom
import xml.dom.ext
 
import kwdebug
MyDebug = kwdebug.Debug(0)
 
class Global_values_Class:
 
    import kwgetopt
    cmd_line_opts = kwgetopt.kwgetopt("a:d:i:o:")
    # Remember: the colon after the option letters mean that there is an argument after those options
    # -a = account containing unbalanced transactions
    # -d = description / account dictionary file
    # -i = input file
    # -o = output file
    #imbal_acct_re = cmd_line_opts.getopt("-a", r"(?i)imbalance.usd") # The other one is "Unspecified" --make sure it doesn't accidentally refer to "Cu Cdn unspecified", by capitalizing "Unspecified")
 
    descr_acct_fname = cmd_line_opts.getopt("-d", "descr_acct.py")
    input_fname = cmd_line_opts.getopt("-i", "gnucash.xml")
    output_fname = cmd_line_opts.getopt("-o", "gnucash-out.xml")
 
    input_unzipped_fname = input_fname + "_unzipped"
    input_unzipped_nocolons_fname = input_unzipped_fname + "_nocolons"
    output_unzipped_fname = output_fname + "_unzipped"
    output_unzipped_nocolons_fname = output_unzipped_fname + "_nocolons"
 
    preamble=''' GnuCashFIX (with transaction description notes, pair matching and RegExp lookup; updated 954JHq)
: This matches corresponding transactions in Gnucash from a designated account.
: It then takes all the transactions in that account
: and moves them to the appropriate accounts based on the description of
: the transaction.
: (-i) GnuCash input: %s
: (-o) GnuCash output: %s
: (-d) Description -> Account matchfile: %s
: (-a) Account to be fixed will be 'Imbalance-USD'
''' % (input_fname, output_fname, descr_acct_fname )
 
    def __init__(self):
        import os.path
        print(self.preamble)
 
        if os.path.abspath(self.input_fname) == os.path.abspath(self.output_fname) :
            print("\n! *Warning*: output will overwrite the input file %s" % self.input_fname)
        else :
            print(": To force output file to overwrite the input file, use the command-line option '-o %s'" % self.input_fname)
 
        try:
            raw_input("> ENTER to continue, or Ctrl-C to abort")
        except KeyboardInterrupt :
            print("\n! Interrupted by keyboard")
            raise SystemExit
 
Global = Global_values_Class()
 
class GnuCashXML_Class:
 
    # I'll make it so that only this class handles XML, and any other classes do not need to access the XML document at all
 
    acct_denominator_tagname = "act__commodity-scu"
    acct_description_tagname = "act__description"
    acct_id_tagname = "act__id"
    acct_name_tagname = "act__name" # Note that it's abbreviated "act", not "acct", in the GnuCash file
    gnc_acct_tagname = "gnc__account"
    #imbal_acct_re = r"(?i)unspecified"
    #imbal_acct_re = r"Uchq" # for testing only
    split_acct_tagname = "split__account"
    #split_id_tagname = "split__id" # This is not used and may be confused with split_acct_tagname
    split_quantity_tagname = "split__quantity"
    split_value_tagname = "split__value"
    transaction_tagname = "gnc__transaction"
    trn_dateposted_tagname = "trn__date-posted"
    trn_description_tagname = "trn__description"
    trn_id_tagname = "trn__id"
    ts_date_tagname = "ts__date"
 
    # To experiment with XML document in Python IDE, use xml.dom.minidom.parse("/home/kwtm1/finances/gnucash-in.conv")
 
    def __init__(self, in_fname = None):
 
        if None != in_fname:
            MyDebug.debug_msg(": Now converting colons")
            conv_fname = Global.input_unzipped_nocolons_fname
            self.colon_to_double_underscore( in_fname, conv_fname )
            MyDebug.debug_msg(": Now importing %s" % conv_fname )
            self.xmldoc = xml.dom.minidom.parse( conv_fname )
        else:
            self.xmldoc = xml.dom.minidom.Document()
 
        self.acct_dict_AcDict = AccountDict_Class({})
 
    def colon_to_double_underscore(self, in_fname, out_fname):
        conv_cmd = "sed -e 's/:/__/g' %s >%s"
        import os
        os.system( conv_cmd % (in_fname, out_fname) )
 
    def double_underscore_to_colon(self, in_fname, out_fname):
        conv_cmd = "sed -e 's/__/:/g' %s >%s"
        import os
        os.system( conv_cmd % (in_fname, out_fname) )
 
    def convert_acct_xml(self, ac):
        # See convert_trans_xml to see how ridiculously nested the XML is.
        ac_name_contentsnode_val = ac.getElementsByTagName(self.acct_name_tagname)[0].childNodes[0].nodeValue
        ac_id_contentsnode_val = ac.getElementsByTagName(self.acct_id_tagname)[0].childNodes[0].nodeValue
        return Account_Class( ac_id_contentsnode_val, ac_name_contentsnode_val, ac)
 
    def convert_trans_xml(self, tr_Tr):
        # Convert from the transactions themselves back to XML
 
        tr_Tr.node.getElementsByTagName(self.trn_dateposted_tagname)[0].getElementsByTagName(self.ts_date_tagname)[0].childNodes[0].nodeValue = tr_Tr.date
 
        tr_Tr.node.getElementsByTagName(self.trn_id_tagname)[0].childNodes[0].nodeValue = tr_Tr.trn_id
 
        try:
            tr_Tr.node.getElementsByTagName(self.trn_description_tagname)[0].childNodes[0].nodeValue = tr_Tr.description
        except IndexError:
            MyDebug.debug_msg( "Bloody transaction %s had no description so we can't modify that." % tr_Tr ,2)
 
        tr_acctids_list = tr_Tr.node.getElementsByTagName(self.split_acct_tagname) # NOT the self.split_id_tagname which is the id of the split itself, not the accounts involved in the transaction
        tr_values_list = tr_Tr.node.getElementsByTagName(self.split_value_tagname)
        tr_quantities_list = tr_Tr.node.getElementsByTagName(self.split_quantity_tagname) # In all GnuCash transactions I've seen, value = quantity, so we can get the value from either one; but when we modify, then we must update both elements
        tr_num_accts_between = len(tr_acctids_list)
 
        if ( tr_Tr.num_accts_between != tr_num_accts_between ):
            MyDebug.debug_msg( "Funny... I thought transaction %s was between exactly %s accounts, but the XML says it's between %s accounts." % (tr_Tr, tr_Tr.num_accts, tr_num_accts_between ),1)
        else:
 
            if ( tr_num_accts_between >= 1):
                tr_acctids_list[0].childNodes[0].nodeValue = tr_Tr.acct1_Ac.acctid
                tr_values_list[0].childNodes[0].nodeValue = self.derive_valuetext(tr_Tr.value)
                tr_quantities_list[0].childNodes[0].nodeValue= self.derive_valuetext(tr_Tr.value)
 
            if ( tr_num_accts_between >= 2):
                tr_acctids_list[1].childNodes[0].nodeValue = tr_Tr.acct2_Ac.acctid
                tr_values_list[1].childNodes[0].nodeValue = self.derive_valuetext( -tr_Tr.value) # The second value is negative to balance the first.
                tr_quantities_list[1].childNodes[0].nodeValue= self.derive_valuetext( -tr_Tr.value) # The second value is negative to balance the first.
 
    def convert_xml_trans(self, tr):
 
        """
        # The XML is nested to a ridiculous degree. To get the date-posted of the transaction, we would have to do:
        tr_dateposted_list = tr.getElementsByTagName(self.trn_dateposted_tagname)
        tr_dateposted = tr_dateposted_list[0]
        tr_dateposted_date_list = tr_dateposted.getElementsByTagName(self.ts_date_tagname)
        tr_dateposted_date = tr_dateposted_date_list[0]
        tr_dateposted_date_contentsnode_list = tr_dateposted_date.childNodes
        tr_dateposted_date_contentsnode = tr_dateposted_date_contentsnode_list[0]
        tr_dateposted_date_contentsnode_val = tr_dateposted_date_contentsnode.nodeValue
        """
 
        tr_dateposted_date_contentsnode_val = tr.getElementsByTagName(self.trn_dateposted_tagname)[0].getElementsByTagName(self.ts_date_tagname)[0].childNodes[0].nodeValue
        #dateposted = self.extract_date(tr_dateposted_date_contentsnode_val)
        #MyDebug.debug_msg( "Extracted date %s" % dateposted,4)
        # No. There is no need to interpret the date ... yet. Just store the date string as is. When we have to compare, only then do we need to interpret the date string stored in a Transaction.
 
        tr_id_contentsnode_val = tr.getElementsByTagName(self.trn_id_tagname)[0].childNodes[0].nodeValue
 
        try:
            tr_description_contentsnode_val = tr.getElementsByTagName(self.trn_description_tagname)[0].childNodes[0].nodeValue
        except:
            tr_description_contentsnode_val = ""
            MyDebug.debug_msg( "Bloody transaction %s has no description!?" % tr_id_contentsnode_val ,2)
 
        tr_acctids_list = tr.getElementsByTagName(self.split_acct_tagname) # NOT the self.split_id_tagname which is the id of the split itself, not the accounts involved in the transaction
        tr_num_accts_between = len(tr_acctids_list)
 
        if ( 2 != tr_num_accts_between ):
            MyDebug.debug_msg( "Funny... transaction %s (%s) is not between exactly 2 accounts, but with %s accounts." % (tr_id_contentsnode_val, tr_description_contentsnode_val, tr_num_accts_between ),4)
 
        if ( tr_num_accts_between >= 1):
            tr_acctid1_contentsnode_val = tr_acctids_list[0].childNodes[0].nodeValue
            acct1_Ac = self.acct_dict_AcDict.get(tr_acctid1_contentsnode_val, None) # Get the account, or return None if no such account
        else:
            acct1_Ac = None
 
        if ( tr_num_accts_between >= 2):
            tr_acctid2_contentsnode_val = tr_acctids_list[1].childNodes[0].nodeValue
            acct2_Ac = self.acct_dict_AcDict.get(tr_acctid2_contentsnode_val, None)
        else:
            acct2_Ac = None
 
        tr_splitvalue_contentsnode_val = tr.getElementsByTagName(self.split_value_tagname)[0].childNodes[0].nodeValue
        value = self.extract_value(tr_splitvalue_contentsnode_val)
 
        # Now to create an instance of Transaction_Class based on the values above
        return Transaction_Class(tr, acct1_Ac, acct2_Ac, tr_dateposted_date_contentsnode_val, tr_description_contentsnode_val, tr_num_accts_between, tr_id_contentsnode_val, value)
 
    def derive_valuetext(self, value, denominator=100):
        return "%d/%d" % (value*denominator, denominator)
 
    def extract_value(self, text):
        # The value text is something like "12345/100" which means "$123.45", but we have to divide numerator by denominator.
 
        try:
            return eval( text.strip() + ".0")
            # If we don't add ".0", then we will be doing integer arithmetic
        except:
            # Okay, just evaluating didn't work, so now we actually have to decode the thing
 
            import re
            value_re = "(\D|^)(?P<numerator>\d+)/(?P<denominator>\d+)(\D|$)"
            match_result = re.search( value_re, text)
            MyDebug.debug_msg("Extracting value from %s" % text,5)
            if match_result == None:
                return None
 
            numerator = float( match_result.groupdict("0").get( "numerator" ) )
            # Get the text that matches the groupname "numerator" in the regular expression (if can't find it, default to "0") and convert it to a float(ing point number) in preparation for calculating the value of this transaction.
            denominator = float( match_result.groupdict("100").get( "denominator" ) )
 
            MyDebug.debug_msg("Value is %s over %s" % (numerator,denominator),5)
            return (numerator/denominator)
 
    def get_acct_dict(self):
        MyDebug.debug_msg("Creating dict of all accounts", 1)
        acct_dict = AccountDict_Class({})
        acct_list = self.xmldoc.getElementsByTagName(self.gnc_acct_tagname)
        for acct in acct_list:
            acct_dict.add_account( self.convert_acct_xml(acct) )
 
        self.acct_dict_AcDict = acct_dict
 
    def get_all_trans_list(self):
        MyDebug.debug_msg("Creating list of all transactions", 1)
        trn_list = self.xmldoc.getElementsByTagName(self.transaction_tagname)
        all_trans_list = Trans_list_Class([])
        for trn in trn_list:
            all_trans_list.append( self.convert_xml_trans(trn) )
        return all_trans_list
 
    def get_unbal_trans_list(self):
        raise NotImplementedError, "Hey, this function is not going to be used."
        #stub
        return []
 
    def modify_original_trans(self, trans_list_TrList):
        for trans_Tr in trans_list_TrList:
            if trans_Tr.need_to_delete_original():
                MyDebug.debug_msg("I would delete %s." % (trans_Tr) ,4)
                trans_Tr.node.parentNode.removeChild(trans_Tr.node)
                trans_Tr.node.unlink()
            elif trans_Tr.need_to_update_original():
                MyDebug.debug_msg("I would update %s." % (trans_Tr) ,4)
                self.convert_trans_xml(trans_Tr)
            else:
                MyDebug.debug_msg("No changes in %s." % (trans_Tr) ,4)
        #stub
 
    def save_changes(self):
        #stub
        pass
 
    def write_to_file(self, out_fname):
        conv_fname = Global.output_unzipped_nocolons_fname
        MyDebug.debug_msg("Writing to file %s" % conv_fname ,1)
        xml.dom.ext.PrettyPrint(self.xmldoc, open(conv_fname, "wb"))
        MyDebug.debug_msg("Converting to %s" % out_fname ,1)
        self.double_underscore_to_colon(conv_fname, out_fname)
 
class Account_Class():
 
    acctid = None
    acctname = None
    acctnode = None
 
    def __init__(self, acctid=None, acctname=None, acctnode=None):
        self.acctid = acctid
        self.acctname = acctname
        self.acctnode = acctnode
 
    def __eq__(self, other):
        if type(other) == type(None):
            return False
 
        if type(other) != type(self):
            raise TypeError, "%s needs to be of type %s" % (other, type(self))
 
        return self.acctid == other.acctid
 
    def __ne__(self, other):
        return not (self == other)
 
    def __repr__(self):
        return "Account %s, which is named %s and resides in node %s" % (self.acctid, self.acctname, self.acctnode)
 
class AccountDict_Class(dict):
    # The dictionary key shall be the account id; the value corresponding to that key shall be an instance of Account_Class that has info about that account.
 
    def __init__(self, *other_args):
        dict.__init__(self, *other_args)
 
    def add_account(self, account):
        self[ account.acctid ] = account
 
    def lookup_by_re(self, name_re):
        # Yes, partial matches do match (ie. if name_re is "bcd" it will find account "abcde")
        import re
        for acct in self.values():
            match_result = re.search( name_re, acct.acctname )
            if match_result != None:
                break
 
        if match_result == None:
            # We didn't find it. We got here because the loop ended, not because we found it and broke out of the loop
            acct=None
 
        return acct
 
class Criteria_Class:
 
    def __init__(self, criteria):
        import re
 
        self.re_c = None
        self.lower_lim = None
        self.upper_lim = None
 
        re_text = "^$" # Initialize to a regular expression that represents the null string
        if str == type(criteria):
            re_text = criteria
        elif tuple == type(criteria) or list == type(criteria):
            if 3 <= len(criteria):
                (re_text, self.lower_lim, self.upper_lim) = tuple(criteria)[:3]
            elif 2 == len(criteria):
                (re_text, self.lower_lim) = tuple(criteria)[:2]
                self.upper_lim = self.lower_lim
            elif 1 == len(criteria):
                re_text = criteria[0]
            # If 0 == len(criteria_text), then we can just use the default values, so we don't have to do anything.
 
        self.re_c = re.compile(re_text)
 
    def __repr__(self):
        return "<criteria: re_c %s, lower_lim %s, upper_lim %s>" % (self.re_c, self.lower_lim, self.upper_lim)
 
    def fulfills_limit_criteria(self, trn_Tr):
 
        # Based on the description of the transaction, such as "MEGA FUEL",
        # as well as upper/lower limits of the transaction amount,
        # figure out whether a given transaction fulfills the criteria
 
        # The order of these tests matter as we go through trying to decide whether the transaction fulfills criteria
 
        #MyDebug.debug_msg("crit.re_c is %s" % self.re_c,3)
        # First check that there's a valid description in the criterion
        if None == self.re_c:
            return False
 
        # Now see if the criterion description matches that of the transaction
 
        if None == self.re_c.search(trn_Tr.description):
            MyDebug.msg_no_cr("N",4)
            return False
        MyDebug.debug_msg("crit matches %s" % trn_Tr.description,4)
        # Now see if there's a valid upper limit (if there's just a lower limit, the upper limit should already have been set to equal the lower limit)
        if None == self.upper_lim:
            MyDebug.msg_no_cr("-",4)
            return True
        elif trn_Tr.value > self.upper_lim:
            MyDebug.msg_no_cr(">",4)
            return False
        elif trn_Tr.value < self.lower_lim:
            MyDebug.msg_no_cr("<",4)
            return False
        else:
            MyDebug.msg_no_cr("=",4)
            return True
 
class Criteria_list_Class(list):
    # contains these properties:
    # crit_dict = the compiled version of the critera. Key = compiled regexp; Value = the account
    # text_dict = the text of the criteria, in a dictionary, read from file
 
    def __init__(self, descr_fname=None, account_dict=None, *other_args):
        list.__init__(self, *other_args)
 
        if descr_fname != None:
            print(": Reading account description from file %s" % descr_fname)
            try:
                self.text_list = self.eval_py_expr_file(descr_fname)
            except TypeError, errmsg:
                print("\n! ERROR: %s" % errmsg)
                print("! Looks like account description file %s is not a correct Python expression. Most likely there is a comma missing at the end of a line." % Global.descr_acct_fname)
                print("! Try looking for the regexp '^[^#]*\)[^,]*($|#)'")
                print("! (that is, a line where a comma does not occur between the closing parenthesis and the end of line/start of comment)")
                raise SystemExit
            except SyntaxError, errmsg:
                print("\n! ERROR: %s" % errmsg)
                print("! Account description file %s is not a correct Python expression. This is not just a comma missing at the end of a line; some extra character got inserted or something." % Global.descr_acct_fname)
                print("! Try deleting parts of the file and retrying, to narrow down where the problem might be.")
                raise SystemExit
 
        else:
            self.text_list = []
 
        if account_dict != None:
            self.acct_dict = account_dict
        else:
            self.acct_dict = AccountDict_Class({})
 
        MyDebug.debug_msg(": initialized crit list" ,2)
        self.initialize_crit_list()
 
    def eval_py_expr_file(self, fname):
        file = open(fname, 'rb')
        py_expr = file.read()
        return eval(py_expr)
 
    def initialize_crit_list(self):
        import re
 
        try:
            for text_list_element in self.text_list:
                (crit_key, acctname_re) = text_list_element[:2]
                acct_Ac = self.acct_dict.lookup_by_re(acctname_re)
                if acct_Ac == None :
                    print("\n! Unidentified account '%s' --ignoring criteria '%s'" % (acctname_re, crit_key))
                else :
                    criteria_Cr = Criteria_Class(crit_key)
                    self.append( (criteria_Cr, acct_Ac) + tuple(text_list_element[2:]) )
                    # Add the criteria and the account name, but also any other elements in the tuple which might correspond to other features
 
        except ValueError, errmsg:
            print("\n! ERROR: %s" % errmsg)
            print("! Is the account description file %s in the wrong format? As of 2009-05-03, it needs to be a list, not a dictionary." % Global.descr_acct_fname)
            raise SystemExit
 
class Timestamp_Class(solitime.Solitime_class):
 
    def __init__(self, *other_args):
        return solitime.Solitime_class.__init__(self, *other_args)
 
    def formatted_timestamp(self, *other_args):
        return solitime.Solitime_class.formatted_solitime(self, *other_args)
 
class Transaction_Class():
 
    node = None
    acct1_Ac = None
    acct2_Ac = None
    #book_Gn = None
    date = None
    description = None
    num_accts_between = 0
    trn_id = None
    value = None
 
    def __init__(self, node=None, acct1_Ac=None, acct2_Ac=None, date=None, description=None, num_accts_between=0, trn_id=None, value=None):
        self.node = node
        self.acct1_Ac = acct1_Ac
        self.acct2_Ac = acct2_Ac
        self.date = date
        self.description = description
        self.num_accts_between = num_accts_between
        self.trn_id = trn_id
        self.value = value
        self.update_flag = False # If this is None, then the transaction is to be deleted
 
    def __repr__(self):
        return "Transaction, with id %s, for amt$ %s, on date %s, between acct %s and acct %s, covering %s accounts, described as %s. Stored in XML node %s." % (self.trn_id, self.value, self.date, self.acct1_Ac, self.acct2_Ac, self.num_accts_between, self.description, self.node)
 
    def __str__(self):
        return "Transaction %s: on %s, $%s from %s to %s." % (self.trn_id, self.date, self.value, self.acct1_Ac.acctname, self.acct2_Ac.acctname)
 
    def extract_timestamp(self):
        mytimestamp_Ts = Timestamp_Class()
        mytimestamp_Ts.set_time_from_string(self.date)
        MyDebug.debug_msg("Converting date from '%s' to '%s'" % (self.date, mytimestamp_Ts.formatted_timestamp()) ,4)
        return mytimestamp_Ts
 
    def flag_to_delete_original(self):
        self.update_flag = None
 
    def flag_to_preserve_original(self):
        self.update_flag = False
 
    def flag_to_update_original(self):
        self.update_flag = True
 
    def need_to_delete_original(self):
        return ( None == self.update_flag)
 
    def need_to_update_original(self):
        return ( True == self.update_flag)
 
class Trans_list_Class(list):
 
    def __init__(self, *other_args):
        list.__init__(self, *other_args)
 
    def find_matching_trans(self, unbal_trans_Tr):
        #found_it = False # We don't need this yet
        for poss_matching_trans_Tr in self:
            if self.the_dates_match(unbal_trans_Tr, poss_matching_trans_Tr) and \
                self.the_accounts_and_amounts_match(unbal_trans_Tr, poss_matching_trans_Tr):
                return poss_matching_trans_Tr
 
        return None
 
    def the_accounts_and_amounts_match(self, unbal_trans_Tr, poss_matching_trans_Tr):
        # Two cases: both transactions share the same Acct1, or same Acct2; then first amount should be negative of the second amount
        # Otherwise if the Acct1 of one is the same as the Acct2 of the other, or vice versa, then the amounts should be identical (not negatives of each other)
        if (unbal_trans_Tr.acct1_Ac == poss_matching_trans_Tr.acct1_Ac) or (unbal_trans_Tr.acct2_Ac == poss_matching_trans_Tr.acct2_Ac):
            return unbal_trans_Tr.value == -(poss_matching_trans_Tr.value)
        elif (unbal_trans_Tr.acct1_Ac == poss_matching_trans_Tr.acct2_Ac) or (unbal_trans_Tr.acct2_Ac == poss_matching_trans_Tr.acct1_Ac):
            return unbal_trans_Tr.value == poss_matching_trans_Tr.value
 
    def the_dates_match(self, unbal_trans_Tr, poss_matching_trans_Tr):
        unbal_trans_timestamp = unbal_trans_Tr.extract_timestamp()
        poss_matching_trans_timestamp = poss_matching_trans_Tr.extract_timestamp()
 
        return_val= abs(unbal_trans_timestamp.time_tuple[0]-poss_matching_trans_timestamp.time_tuple[0]) <= 1 and \
            unbal_trans_timestamp.time_tuple[1] == poss_matching_trans_timestamp.time_tuple[1] and \
            unbal_trans_timestamp.time_tuple[2] == poss_matching_trans_timestamp.time_tuple[2]
        # The dates will be considered matching if the month and day match exactly, and if the years match or are exactly 1 apart (my own special code)
        return return_val
 
class Transfix_Class():
 
    def __init__(self):
        pass
 
    def classify_trans_by_descr(self, trn_Tr):
        # Based on the description of the transaction, such as "MEGA FUEL",
        # figure out which income/expense account this should be, such as "Expenses:consumables:gasoline"
        #description_node = trn.getElementsByTagName(Global.trn_description_tagname)
        #if 0 >= len(description_node):
        # raise AccountError, "Can't find transaction description"
        #description = description_node[0].childNodes[0].nodeValue
 
        return_acct_Ac = None
        return_trans_descr = None
        for element in self.criteria_list_CrList:
            (criterion_Cr, acct_Ac) = element[:2]
            if criterion_Cr.fulfills_limit_criteria( trn_Tr ):
                # The transaction description matches, so let's see what account this transaction should be under
                return_acct_Ac = acct_Ac
                if len(element) >=3:
                    return_trans_descr = element[2] + " - " + trn_Tr.description
                    # ToDo: change this to a more sophisticated substitution
                break
 
        return (return_acct_Ac, return_trans_descr)
 
    def classify_trans_of_translist_by_descr(self, remaining_unbal_trans_list_TrList):
 
        print("")
        for unbal_trans_Tr in remaining_unbal_trans_list_TrList:
            (corrected_acct_Ac, corrected_trans_descr) = self.classify_trans_by_descr(unbal_trans_Tr)
            if None != corrected_acct_Ac:
                # Do this only if we've actually found the corrected acct id, or else leave the acct id alone
 
                # But is it acct1 or acct2 of the transaction that we need to correct? The following takes care of this.
                if unbal_trans_Tr.acct1_Ac in self.unbal_acct_list:
                    unbal_trans_Tr.acct1_Ac = corrected_acct_Ac
                elif unbal_trans_Tr.acct2_Ac in self.unbal_acct_list:
                    unbal_trans_Tr.acct2_Ac = corrected_acct_Ac
 
                if corrected_trans_descr != None:
                    unbal_trans_Tr.description = corrected_trans_descr
 
                unbal_trans_Tr.flag_to_update_original()
                MyDebug.msg_no_cr("!", 8)
                print(": (%s) $%0.2f\t%s -> '%s'" % (unbal_trans_Tr.date, unbal_trans_Tr.value, unbal_trans_Tr.description, corrected_acct_Ac.acctname))
            else:
                print("! (%s) $%0.2f *\t*\t* no account found for '%s'" % (unbal_trans_Tr.date, unbal_trans_Tr.value, unbal_trans_Tr.description))
 
    def extract_unbal_trans_list(self, all_transactions, unbal_acct_list):
        # First, clean up unbal_acct_list because it might contain None's
        no_more_Nones = False
        while not no_more_Nones:
            try:
                unbal_acct_list.remove(None)
            except:
                # We couldn't remove any more Nones from the list, so we must have gotten rid of all of them
                no_more_Nones = True
 
        MyDebug.debug_msg("After removing Nones, our unbal_acct_list is %s" % unbal_acct_list,4)
 
        unbal_trans_list = Trans_list_Class([])
        for tr in all_transactions:
            MyDebug.msg_no_cr("t", 3)
            #MyDebug.debug_msg("Checking tr %s." % tr,4)
            for unbal_acct in unbal_acct_list:
                MyDebug.debug_msg("Checking with account %s." % unbal_acct,4)
                if (tr.acct1_Ac == unbal_acct) or (tr.acct2_Ac == unbal_acct):
                    MyDebug.debug_msg("Found that transaction %s is unbalanced." % tr, 3)
                    unbal_trans_list.append(tr)
 
        return unbal_trans_list
 
    def find_matching_pairs_of_unbal_trans(self, unbal_trans_list_TrList):
    #def get_book_fn(self):
    # return "gnucash.xml"
        self.matched_unbal_trans_list = [] # This will contain tuples (pairs) of the matched accounts
        self.unmatched_unbal_trans_list_TrList = Trans_list_Class([])
        self.already_matched_unbal_trans_list_TrList = Trans_list_Class([])
        for unbal_trans_Tr in unbal_trans_list_TrList:
            if unbal_trans_Tr not in self.already_matched_unbal_trans_list_TrList :
                match_unbal_trans_Tr = unbal_trans_list_TrList.find_matching_trans(unbal_trans_Tr)
                if None != match_unbal_trans_Tr:
                    # We found a match! Now put the match in already_matched_unbal_trans_list_TrList so we don't try to look for a match (which would produce duplicate match)
                    self.already_matched_unbal_trans_list_TrList.append( match_unbal_trans_Tr )
                    self.matched_unbal_trans_list.append( (unbal_trans_Tr, match_unbal_trans_Tr) )
                    #my_gnucash_book_Gn.reconcile(unbal_trans_Tr, match_unbal_trans_Tr)
                    MyDebug.debug_msg("Found match. Currently %s unbalanced and %s pairs of matched transactions." % ( len(self.unmatched_unbal_trans_list_TrList),len( self.matched_unbal_trans_list ) ) ,4)
                else:
                    self.unmatched_unbal_trans_list_TrList.append( unbal_trans_Tr )
                    MyDebug.debug_msg("Could not find match for %s" % unbal_trans_Tr,4)
                    MyDebug.debug_msg("No match. Currently %s unbalanced and %s pairs of matched transactions." % ( len(self.unmatched_unbal_trans_list_TrList),len( self.matched_unbal_trans_list ) ) ,4)
        print(": Could not find match for %s unbalanced transactions." % len(self.unmatched_unbal_trans_list_TrList))
        print(": %s pairs of transactions to be reconciled." % len( self.matched_unbal_trans_list ))
 
        for (tr1_Tr, tr2_Tr) in self.matched_unbal_trans_list:
            self.reconcile_matching_unbal_trans(tr1_Tr, tr2_Tr)
 
        return self.unmatched_unbal_trans_list_TrList
 
    def reconcile_matching_unbal_trans(self,tr1_Tr, tr2_Tr):
        # The goal is to modify the transaction where the *second* account (not the first) is the shared one.
        # Then we delete the transaction where the first account is the shared one. So, if we have:
        # tr1: $10 from Acct A to Acct B
        # tr2: $10 from Acct B to Acct C
        # Then we delete tr2, and change tr1 to: $10 from Acct A to Acct C
 
        if (tr1_Tr.acct2_Ac == tr2_Tr.acct1_Ac):
            tr1_Tr.acct2_Ac = tr2_Tr.acct2_Ac
            tr2_Tr.acct1_Ac = tr1_Tr.acct1_Ac
        elif (tr1_Tr.acct2_Ac == tr2_Tr.acct2_Ac):
            tr1_Tr.acct2_Ac = tr2_Tr.acct1_Ac
            tr2_Tr.acct2_Ac = tr1_Tr.acct1_Ac
        elif (tr1_Tr.acct1_Ac == tr2_Tr.acct2_Ac):
            tr1_Tr.acct1_Ac = tr2_Tr.acct1_Ac
            tr2_Tr.acct2_Ac = tr1_Tr.acct2_Ac
        elif (tr1_Tr.acct2_Ac == tr2_Tr.acct2_Ac):
            tr1_Tr.acct2_Ac = tr2_Tr.acct1_Ac
            tr2_Tr.acct2_Ac = tr1_Tr.acct1_Ac
        else:
            raise AssertionError, "Looks like these aren't really matching transactions: %s and %s" % (tr1_Tr, tr2_Tr)
 
        # At this point each of the two transactions where one of the accounts was the Imbalance account has had its Imbalance account replaced with the correct account. So there are 2 duplicate transactions and we need to get rid of one.
        # Which do we keep? The one with the earlier date, since my special code is that I will put a date one year ahead. That is, if I deposit a cheque for $123.45 on 2003-06-21, then I'll put a reminder transaction that on 2004-06-21, $123.45 went somewhere. So even if the cheque doesn't show up right away, I will have a reminder that there is a transaction there... somewhere. I think. Does that work? Whatever. Just do it.
        if tr1_Tr.date < tr2_Tr.date :
            tr1_Tr.flag_to_update_original()
            tr2_Tr.flag_to_delete_original()
        else:
            tr2_Tr.flag_to_update_original()
            tr1_Tr.flag_to_delete_original()
 
        MyDebug.debug_msg("Reconciled transactions to: \n%s, and \n%s" % (tr1_Tr, tr2_Tr) ,4)
 
    def xml_manip_main(self):
        import os
 
        print(": Reading from finances file '%s'" % Global.input_fname)
        MyDebug.debug_msg("Now unzipping")
        result = os.system( "gunzip --to-stdout %s >%s" % (Global.input_fname, Global.input_unzipped_fname) )
        MyDebug.debug_msg("unzip result is %s" % result)
        if result != 0 :
            print(": %s does not appear to be zipped. Trying as unzipped file." % Global.input_fname)
            input_unzipped_fname = Global.input_fname
        else :
            input_unzipped_fname = Global.input_unzipped_fname
 
        self.my_gnucash_book_Gn = GnuCashXML_Class(input_unzipped_fname)
        self.my_gnucash_book_Gn.get_acct_dict()
        self.acct_dict_AcDict = self.my_gnucash_book_Gn.acct_dict_AcDict
        #MyDebug.debug_msg("Formed account dictionary of size %s" % len(acct_dict_AcDict) ,1)
        print(": Found %s accounts in %s" % (len(self.acct_dict_AcDict), Global.input_fname) )
        MyDebug.debug_msg(self.acct_dict_AcDict,4)
 
        self.all_trans_list_TrList = self.my_gnucash_book_Gn.get_all_trans_list()
        #MyDebug.debug_msg("There are %s transactions in total." % len(all_trans_list_TrList) ,1)
        print(": Found %s transactions in %s" % (len(self.all_trans_list_TrList) , Global.input_fname) )
 
        self.unbal_acct_list = [self.acct_dict_AcDict.lookup_by_re("Unspecified"),
                    self.acct_dict_AcDict.lookup_by_re("Imbalance-USD")]
        MyDebug.debug_msg("Unbal acct list is %s" % self.unbal_acct_list,2)
 
        self.unbal_trans_list_TrList = self.extract_unbal_trans_list( self.all_trans_list_TrList, self.unbal_acct_list )
        print(": There are %s unbalanced transactions." % len(self.unbal_trans_list_TrList))
 
        # We will store the original unbal_trans_list,
        # and then do various things like match up corresponding transactions and switch the account to the right account based on the description
        # Each time we do this, we maintain a list of transactions that are still not fixed, and pass it to the next function
 
        # Step 1
        #remaining_unmatched_unbal_trans_list_TrList = self.unbal_trans_list_TrList
        remaining_unmatched_unbal_trans_list_TrList = self.find_matching_pairs_of_unbal_trans(self.unbal_trans_list_TrList)
        #MyDebug.debug_msg(": skipping matching pairs. Revert this later on.")
 
        #remaining_unmatched_unbal_trans_list_TrList = self.fix_account_if_needed_based_on_descr(remaining_unbal_trans_list_TrList)
 
        # Step 2
        self.criteria_list_CrList = Criteria_list_Class(Global.descr_acct_fname, self.acct_dict_AcDict)
        remaining_unmatched_unbal_trans_list_TrList = self.classify_trans_of_translist_by_descr(remaining_unmatched_unbal_trans_list_TrList )
 
        self.my_gnucash_book_Gn.modify_original_trans( self.unbal_trans_list_TrList )
 
        print(": Done. Now writing to file '%s'" % Global.output_fname)
        self.my_gnucash_book_Gn.write_to_file(Global.output_unzipped_fname)
        MyDebug.debug_msg("Now zipping")
        result = os.system( "gzip --to-stdout %s >%s" % (Global.output_unzipped_fname, Global.output_fname) )
        MyDebug.debug_msg("zip result is %s" % result)
        if result != 0 :
            print(": %s may not have been zipped properly. Try using unzipped intermediate file %s" % (Global.output_fname, Global.output_unzipped_fname) )
 
def main():
    mytransfix = Transfix_Class()
    mytransfix.xml_manip_main()
 
if __name__ == "__main__":
    main()

End of listing for my Python program.

User Journal

Journal: detecting subtle changes in Terms & Conditions

Journal by KWTm

This is to check for subtle changes in text that you see often, such as Terms & Conditions that pop up every time you use a frequently-used Web service. It will alert you to small changes that you might not otherwise notice because you habitually click on the "I Agree To These Terms And Conditions" button without going through all the text each time. Simply select the text and copy to the clipboard, and then run the script.

This shell script compares what's in the clipboard to text files in a certain directory; in this case it's ~/Documents/terms_and_conditions. This is where I would store the T&C text from various web sites. When it finds a similar match, it does a diff to look for minor changes. If there is no exact match, it offers to store the copied text in a new text file so you can compare the current version to future versions.

It works with the KDE clipboard, Klipper. Those of you who use GNOME or some other clipboard system will need to modify the line that says: "dcop klipper klipper getClipboardContents >$TEMP_FILE". (In fact, I think those of you using KDE 4 will need to modify this line, too.)

#!/bin/sh
 
COMPARISON_DIR="$HOME/Documents/terms_and_conditions"
COMPARISON_LENGTH=160
SAVE_FOR_FUTURE_FN="clipboard_saved"
TEMP_FILE="temp.tmp"
TEMP_FILEHEAD="temp_head.tmp"
TEMP_CANDIDATE_HEAD="temp_cand_head.tmp"
 
QUIET_FLAG=`echo " $@" | grep -cE " -Q"`
IMMEDIATE_FLAG=`echo " $@" | grep -cE " -I"`
 
if [ $QUIET_FLAG -le 0 ] ; then :
    # The Quiet cmd line argument was NOT specified.
    echo " "
    echo " CompareTermsAndConditions [options] [-I] [-Q] (cannot specify '-IQ' together; '-I -Q' is ok. )"
    echo ": This compares what's in the KDE klipboard with files in $COMPARISON_DIR"
    echo ": automatically searching all files to see which file might be similar or identical."
    echo ": This is to look for changes in those Terms & Conditions on web sites which show them repeatedly, to make sure that after you stopped bothering to check for changes, they don't sneak in unnoticeable changes to the Terms & Conditions."
    echo ": Make sure you have copied the text to clipboard."
    echo ": '-Q' will suppress output to stdout; '-I' will skip the prompt and proceed immediately"
    if [ $IMMEDIATE_FLAG -le 0 ] ; then :
        # The immediate line argument was NOT specified.
        read -p "> ENTER to continue, or Ctrl-C to abort" RESULT
        echo "\n"
    fi
fi
 
OLD_DIR=$PWD
cd $COMPARISON_DIR
 
dcop klipper klipper getClipboardContents >$TEMP_FILE
PATTERN=`grep --max-count=1 '[A-Za-z]' <$TEMP_FILE`
echo ": We will look for files that start with '$PATTERN'"
FILELIST=`grep --fixed-strings -- "$PATTERN" * | sed -r -e 's/([^:]*):.*$/\1/' | uniq`
echo ": Will compare your text to the following similar-looking files:"
echo $FILELIST
 
FOUND_IDENTICAL="False"
for FN in $FILELIST; do :
    if [ $TEMP_FILE != $FN -a $TEMP_FILEHEAD != $FN -a $TEMP_CANDIDATE_HEAD != $FN ]; then :
        #echo " "
        head --bytes=$COMPARISON_LENGTH $TEMP_FILE > $TEMP_FILEHEAD
        head --bytes=$COMPARISON_LENGTH $FN > $TEMP_CANDIDATE_HEAD
        diff --ignore-all-space --brief $TEMP_FILEHEAD $TEMP_CANDIDATE_HEAD > /dev/null
        if [ $? -eq 0 ]; then :
            # Yes, $FN is really worth comparing.
            # We check this just to make sure it's not spuriously matching blank lines in the cliptext.
            echo ": Now comparing with $FN"
            diff --report-identical-files --ignore-all-space $TEMP_FILE $FN
            if [ $? -eq 0 ]; then :
                FOUND_IDENTICAL="True"
                echo ": --- IDENTICAL! ---"
                echo " "
            fi
        else :
            echo ": Skipping $FN, which isn't really all that similar."
        fi
    fi
done
 
if [ "False" = $FOUND_IDENTICAL ]; then :
    echo " "
    if [ $IMMEDIATE_FLAG -le 0 ] ; then :
        # Immediate flag was NOT specified, so we can be interactive and ask if wants to save text to file for future comparison
        echo "! NO MATCH. Want to save the current text for future comparison? (Specify '-I' next time to automatically say yes.)"
        read -p "> ('y'=yes, else no):" RESULT
        if [ "a_y" != a_$RESULT ]; then :
            exit # Otherwise proceed to save
        fi
    fi
    TIMESTAMP=`date +%y%m%d%H%M%S`
    mv $TEMP_FILE "$SAVE_FOR_FUTURE_FN-$TIMESTAMP.txt"
    echo ": Now saving to file $PWD/$SAVE_FOR_FUTURE_FN-$TIMESTAMP.txt"
fi

User Journal

Journal: new OpenPGP key as of 2008-08-01

Journal by KWTm

Well, gee, the plan was to sign this new key before the old key lapsed, but lazy me... anyway, at least I'm still using the same user account. The idea is that, if I get kicked out of this account (or someone takes over this account), then I can set up a new Slashdot account and use this key to prove that I'm the same user.

Here it is. In case Slashdot formats it, remember that if there are any extra-long lines, those should be broken up into individual lines. Each line of this OpenPGP key is the same length and contains no spaces.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.6 (GNU/Linux)

mQGiBEgo38YRBADIxPJE2WJjqCnqp7dshy772sOfpdud15I+pJSRVq8/ED8R6/A/
NFe8GCqL5//IRvj2WaJtd8OLVfJtzb+4Q8kTvgtsBCQU02KyBGEBGhyfxwUBxEYP
Ynwu2GdhkodLRKOEdmLuegvirmxbjFr2Bb408lRG7UkUbWH2uZRTvIoRYwCg8Pib
56ZSgLIRZbip/MLnavMtrKUD/RHCrUDJg09zmN9P0DwaFdod1Tp5xNg/ko3uGli9
0+WjPEMCQCK51nwNEnPijhbRq38f/Jbut/fFJlawwEyrEaufIfGr+FwLX2zKhgSM
a57q/b4dzz9f7Ky4vwoJK+Oh8zlISL+uZs5RPRep+GjYUJTjMa8D+R9txb7gX/tX
17coBAC7Akp8SFiXvbylN1eSzvWUYiwoD7R9m4naLhHpIcf1A6J1H1Tra/ZvsMcp
k+Cz6gy3FrZn2Az6Gi2+gvJE04ma0loKN5kN90tNm9BEM24/25Ontef1ubL47Dty
iY8MNfTFzcTCaapmwGJQ4h017p5Z8hyXFglz4Stdpa7CgK1GsLQ2S2EgV2FpIFRh
bSAoMjAwOCBNYXkgMTIgS21haWwpIDxrYXdhaS1rYkB0YW1seWxpbi5uZXQ+iGYE
ExECACYFAkgo38YCGyMFCQPCZwAGCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRBx /pIpG5px4O6DAJwNbNQEUakeF6mPRb3Q4T18ZAP1YgCg5LSpeC/tO3SI8g9X7n4F
4K2VGGS5AQ0ESCjfxxAEAKDFoVXqAAOI1LMetFiSqbBxc4cf813955ltLY88qJi8
1bJTWFQe1YFl4Kq/i71gkri9xlkNnERemipsyoHv/i2Pb1FeQvzLm6BnvZqRCqpQ
eRg4TOcaoQ/kFecBTBg4rGwVXG25Jjffr336LJ0/cO0LQGcZxRadAwv3l50XYubX
AAMHBACWK80MNxV9bVjO9RwY7DHfchETq7NtLVFwxjkdPacROfqQSEJ0i1c1eJ+Y
wPt22i9B3tT0g26wtqexKp0GAE6FP0IHrrw8QeI1BEkepvF3IPfVspJNJ89/0e3u
Wvkk7n6QtRbfYKp2M+V6WSh76TgtUdCxG7xNxsQXQYu5e+cK24hPBBgRAgAPBQJI
KN/HAhsMBQkDwmcAAAoJEHH+kikbmnHg/x0AoKRNFGP+WKUZxehMIzutH6lPWr78
AJ9K+Ng5s0mUCvPX1PmnDJG6uqkI4A==
=oRdN
-----END PGP PUBLIC KEY BLOCK-----

User Journal

Journal: My Dellbuntu laptop: installing Kubuntu

Journal by KWTm

Installing Kubuntu:
- the plan is to shrink the built-in Ubuntu partition on /dev/sda6 down to 6GB (my original plan was 4GB, but there's a frigg'n 4.3GB of files in there already! What did they load on there, some voice-recognition stuff?)
- expand the swap partition to 4GB (since there's 2GB of RAM in there)
- then add another 4GB partition to install Kubuntu
- then the remaining 130+ GB is for data

- running from the Kubuntu 7.04 Live DVD, QTParted is unable to resize the main 151GB partition! --can't move, can't resize; I can only choose to delete it if anything. The partition was not mounted, so not sure why I couldn't manipulate it.
- so we need some other tools for managing it, instead of QTParted. I bought PartitionExpert from Acronis (in 2003, it was $45, downloaded from web), which is able to handle Ext3 as well as ReiserFS partitions. At the time, Partition Magic (which was v6 at the time) was not able to do ReiserFS. Anyway, so in this step I was not able to move ahead with any FLOSS disk partition manager known to me. (Any suggestions for other partition managers I should try, in the future?)
- using Partition Expert: surprise! The 4.3GB of data turned out to be only about 1.7GB, taking up less room than I thought. I was able to shrink the gUbuntu partition (/dev/sda6) down to 4GB, create a new 4GB partition for Kubuntu (/dev/sda7), and make the remaining 135GB partition (/dev/sda8) for data.

(Here are partial results for "df -h", listing device, total size, used size, remaining size, %used, and mount point.) /dev/sda1 47M 876K 47M 2% /media/sda1 /dev/sda2 2.0G 693M 1.4G 34% /media/sda2 /dev/sda3 193M 21M 163M 12% /media/sda3 /dev/sda6 4.0G 1.7G 2.1G 45% /media/sda6 /dev/sda7 4.0G 2.8G 1000M 74% / /dev/sda8 135G 3.4G 132G 3% /media/sda8
(/dev/sda4 is not listed. That's the swap partition, which I increased to 4GB.)

- it took a long, LOOOooooo..ng time to do it, though. Using Partition Expert, I first clicked on the "big" partition (at the time it was /dev/sda6, 150GB), just to say, "I'll work with that partition". A window popped up: "analyzing partition", and then it froze. At least, it looked like it. Actually, it really was analyzing the partition, except it took 20 minutes! So, beware not to click on the wrong partition by accident.
- and then once I specified how I wanted the partitions set up, it took another 40 minutes or so. Not sure --it took so long that I lost track.

- Once everything was resized, Kubuntu installed smoothly.
- Previously, installing Kubuntu on my desktop boxes, I did very little manually; instead, I typed commands into a script and then ran the script, eventually accumulating a long record of my installation steps in a shell script. This paid off now as I copied and pasted large chunks of this script into a similar installation script for the Dellbuntu, and it took only about half a dozen steps (with big sections of nothing but "sudo apt-get --assume-yes install" lines) to reproduce my desktop configuration on the Dellbuntu.
- by installing the 915resolution package ("sudo apt-get install 915resolution"), I got to use the full resolution of 1280x800 \
- I previously worried about the screen being too small, but I guess with the full resolution the screen looks pretty big. I'm rather pleasantly surprised. My desktop has a 19" monitor capable of 1280x1024, but the Linux driver for the onboard graphics card can only drive it at 1024x768. I never realized till now how much screen size is dependent on resolution rather than just physical size.
- By the way, I chose the normal matte screen rather than the default Dell TrueLife(tm) Screen With Sharper Colours And Annoying Reflection. I'm glad! The screen is readable in a variety of lighting conditions.
- while on the subject of hardware, the keyboard feels nice. The one minor thing I have to get used to is the mousepad: occasionally, one of my thumbs accidentally touch it while I type, and the computer thinks I clicked on the mouse button.

Next step: installing Beryl! (Of course! That's the whole point of getting a laptop with Linux on it --to show off! :) )

User Journal

Journal: Dellbuntu laptop arrived! 1

Journal by KWTm

Today, the Dell laptop (Inspiron 1505n) with pre-installed Ubuntu arrived today. I will post my review blog as replies to my previous journal entry about Buying A Dellbuntu so that all the comments and threads can be organized into one place.

News flash: Nope, I won't be posting it to my previous journal entry because Slashdot has archived that entry and its replies, and no one can make changes. What the tutti fruitti!?? Okay, fine, I'll post the review as a reply to here.

User Journal

Journal: Buying a Dellbuntu 7

Journal by KWTm

I'm checking out the journal system on Slashdot. Hmm, it looks like I can enable and disable comments for individual journal entries. That's good.

Anyway, in this post about buying a Linux computer from Dell, I said I would post in my journal about how the purchase was going. Knowing me, I'll probably be too lazy to post much, but I figured that posting this in my journal would let others comment without having to post directly under the main thread.

Here's a copy of what I had posted:

I thought I should hang back and let others do the initial buying, to see how well this works out and whether the hardware crashes and burns. But if everyone did that, then nobody would buy because no one would want to be first. Since I've been looking forward to getting a Linux notebook, I think it should be okay for me to be one of the first "tryer-outers". Also, hopefully this venture of Dell's into Ubuntu will be high-profile enough that if I encounter any problems, I'll scream and shout that I'm going to post about my problems on Slashdot, and then Dell shall suffer the wrath of Slashdot!! and they'd be more willing to fix it.

In addition to the basic notebook at $599, I decided to upgrade the memory from 512MB to 2GB (+$200), since it's probably the most precious commodity around; if I try to upgrade later, say in 2 years, some new memory standard will probably have come out and I won't be able to find the proper chips.

I figured I'd upgrade the hard drive, too, from 80GB to 160GB. I had thought I would upgrade the 2.5" HDD myself, but it comes with a SATA hard drive, and I've only worked with PATA hard drives[1]. Anyway, that's another +$125 for the HDD upgrade.

My third upgrade is for the DVD burner. The original price comes with a CD burner/DVD-ROM drive, but I've always had problems with Linux and DVD burning --my Kubuntu box has the LITE-ON DVD DL burner, and so far I've had to power up our Win2k box to burn DVD's. For +$40, I'm happy to get the DVD DL burner, and I want to see if K3b will let me burn all 8GB+ onto a DL DVD. Would be sweet if I could.

The only thing I don't like is the screen size. I don't care about widescreen[2], and you can't directly compare diagonal screen sizes of 16:9 (widescreen) screens with 4:3 (conventional) sizes, so I converted. The diagonal of a 16:9 screen is 1.22 times as long as a 12:9 (that is, 4:3) screen for the same height, so I divided the 15.4" diagonal length of the widescreen by 1.22 to get 12.6". So I'm really getting a 12.6" screen, except it's wider. That's tiny. The ThinkPad that my work gives me is 15" (4:3 aspect, same screen height as 18.3" widescreen) and I don't think it's big enough. Well, at least the small screen size makes the laptop smaller and portable.

By the way, what the heck is "TrueLife (glossy)"? I have the option to have it or not have it for my screen, at the same price, but it sounds like a load of MarketSpeak.

So, anyway, here's my system, cut&pasted from the Dell page:

Intel® Pentium® dual-core proc T2080(1MB Cache/1.73GHz/533MHz FSB
Ubuntu Edition version 7.04
15.4 inch Wide Screen XGA Display with TrueLife(TM)(glossy)
2GB Shared Dual Channel DDR2 SDRAM at 533MHZ, 2 DIMM
160GB 5400 RPM SATA Hard Drive
8X CD/DVD Burner (DVD+/-RW) with double-layer DVD+R write capability

53 WHr 6-cell Lithium Ion Primary Battery
Intel PRO/Wireless 3945a/g

1Yr Ltd Warranty and Mail-In Service
Recycling Kit and Plant a Tree for Me

Intel® Graphics Media Accelerator 950
Integrated Audio
Intel Centrino Core Duo Processor

I'll probably sit on this till next week, and then make the purchase.
Any comments? Is this a good deal, or am I being foolish?

I'm experimenting with the Slashdot journal, so maybe I'll post stuff in my journal [slashdot.org] about how the purchase is going, and I think I can set it up so that people can post comments.

-----
[1] PATA notebook drives: It's not that I'm afraid of SATA drives; it's that I've been standardizing on PATA 2.5" drives because I have a number of 2.5" notebook enclosures that, for $25, turn the internal notebook HDD into an external USB HDD that fits into my shirt pocket.

[2] widescreen: Please don't give me that crap about "But if you're screen's not wide enough, you don't see the whole movie --it will be chopped off at the left and right sides!" Well, then, just shrink the movie! I don't see anyone ever saying, "You need a 4:3 screen, because your TV show will be chopped off at the top and bottom by a 16:9 screen!"

User Journal

Journal: My OpenPGP key

Journal by KWTm

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.2.4 (GNU/Linux)

mQGiBERe4zsRBACadO3RJcTnmyzjemGXqXuwbVJtSkijmHaB5nlSvn/rXpbk5H9l
GKK1Dog3V6x74Pj0AwTyaZLRpKT1pjk4MRv7zFnw2W3ZTxbnVduw7n+DnIRWZr38
2TRWUR96BlDwC6iUDwvAqosv+m5YrUvnuuLvOxTyfdt5OQWvHUOnfSnWiwCglnGq
GQtGa09nikCpQDi3l4e/lgMD/0e8MXWOYNZAMbSA1fQs8ZgpM1+Ll2r8/MZ8KGjM
brUhdcRx4gl62/9mq4as5R/1OURs2OvWkrSYfv/jdHRoZwiYAYsY0Jef7O58vqfK
OaOELG3mITh21fmcxrbCqO/0IgdlVC0kTWUNUBWAm1y1jE87SHly2AOIbOHUdwN4
Ads7A/9XWPvrT2Z5d7QWnT/45YqYRSJdyN8MuAuUPz580Xd4JYht4lR65SUB+yU/
kyKHnfZ0smg9MfKnbMcStTiqOGV1DTPsPClWtcI371QNbJxWuhFsStjJNT0yuCBC
vMs5lYV6UdFhcnI+T749VuWPd2foCNuXo0M/GV0yoI5W0GFMb7Q2S2EgV2FpIFRh
bSAoMjAwNiBNYXkgIkdudSBQRyIpIDxrYXdhaS1rYkB0YW1seWxpbi5uZXQ+iGEE
ExECACEFAkRe4zsFCQPCZwAGCwkIBwMCAxUCAwMWAgECHgECF4AACgkQB8iUW0ha
VSkqbQCeKecVPYM/L/P+A7oPg695Ne7mAJ8AniODa07BU4KgJ/xp8sKviQfOEt54
uQINBERe40QQCADdRZSrU/yAvsfIAf457k1xGETGsnZAPozzRe95KSqLMxAjeQ04
O4SxZYP6YuNEnE6J60VFfrvyv/uKxJtB3SRXYCwL1WG83LPcdEr6HMtRfyBH+fcA
sMTVuPKOfZqRcRJmKu9C1i1IjGgaT65bDJWHxHmWsnM3lfzIR4iOIaDDnQLwGogR
SYk7DG6jggoixi+tNRlssntl91iyxA8P5L9H08xih2bwXR3RGjwFBM1y/lBlPWZy /Vwf0FqUVas9GwVqvQytCTHnzigK1XUam2Q0NYg+7TiUn0NssJklcyMcD/oVTStd
lFCAYnRUzn/M3XB3Rwn4mp1kNAr0ldKP1ZJ3AAMFB/9LNRwhCaOFt6FI7WYdJrZr
1k+98RoktIL42Sm6TuMK2r4w/jtnby2gMMhOctnIdQsLgpjpYUJLQz9Q3K91Gw/k
8CwDrKJKzuTXgu9ds+HERx/3qMTVrWKXQ93XOBRDjd7iw2VqNKPQcc1/D+VXNIr4 /FReanrPvlrAjeTJC+Evk3HabfHgtmCj9HOrlj0FSanHjbunO4D0i+Bk4ZXPVsb2
vNPY14niiHnZFXefhXq2OUoy16fND0IZ3wYndslcjnar/bPvtTJ/KVeRMvxjGgQw
cNsxuxe2y5Iu5IAGpHvG/mnuGCMnjO4s2mTMfikVRTOQdKeTg9cocrVauEjq+jxG
iEwEGBECAAwFAkRe40QFCQPCZwAACgkQB8iUW0haVSlEjgCeN0qzjK49n2q1Wr+8
cUhtJLj+UG0An22+XOPL1TIwf3LZfyc6y1pTSp45
=nLy2
-----END PGP PUBLIC KEY BLOCK-----

Prediction is very difficult, especially of the future. - Niels Bohr

Working...