馬克雜想...

I tried to figure out how to use Hadoop and HDFS for a while. But the information on their official site is very scattered and out-of-updated. Here are some notes I did. A clean ubuntu 10.04 LTS build. Download hadoop package from here . The Hadoop's versioning rule is very confusing. 1.0.x is stable version 1.1.x is beta version 2.x.x is alpha version 0.23.x is similar to 2.x.x but missing Name Node HA I tried to ignore all other version started with 0.2x. I just use 1.0.4 directly. Download the KEYS in the root directory Download the from hadoop_1.0.4-1_i386.deb(or it's x64 version) and it's asc file from hadoop-1.0.4 folder. Check the integrity run `gpg --import KEYS` run `gpg --verify hadoop_1.0.4-1_i386.deb.asc` You should see mac@mac-ubuntu:~/projects/hadoop$ gpg --verify hadoop_1.0.4-1_i386.deb.asc gpg: Signature made Thu 04 Oct 2012 01:04:55 PM PDT using RSA key ID ECB31663 gpg: Good signature from "Matthew Foley (CODE SIGNING KEY) <m

TrueCrypt  is a good stuff that you can encrypt your data to a virtual disk. Which is actually a file resides in your regular file system. And that file can be put in your Dropbox folder, so your data can be stored in "cloud" securely.

Encryption openssl enc -e -in original_file -out original_file.aes -aes256 -k password Decryption openssl enc -d -in original_file.aes -out original_file.out -aes256 -k password AES size: original file size + 1, then padding to 16bytes, then add 16 e.g. 1 117 bytes 117 + 1 padding to 16 bytes => 128 bytes 128 bytes + 16 = 144 bytes e.g. 2 127 bytes 127 + 1 padding to 16 bytes => 128 bytes 128 bytes + 16 = 144 bytes e.g. 3 128 bytes 128 + 1 padding to 16 bytes => 144 bytes 144 bytes + 16 = 160 bytes It's irrelevant to the length of password.

http://lucene.apache.org/core/3_6_1/demo.html CLASSPATH OK export CLASSPATH=/home/mac/xxxx/xxx/xxx.jar:/home/mac/yyyy/yyy/yyy.jar export CLASSPATH=/home/mac/xxxx/xxx/*:/home/mac/yyyy/yyy/* Not OK export CLASSPATH=/home/mac/xxxx/xxx/*.jar:/home/mac/yyyy/yyy/*.jar export CLASSPATH=/home/mac/xxxx/xxx/:/home/mac/yyyy/yyy/ http://lucene.apache.org/core/3_6_1/demo2.html Need to detect doc language and change to use correct analyzer. Create Index open an directory to put index files (dir) new an Analyzer (analyzer) new an IndexWriterConfig (iwc) do some settings on IndexWriterConfig use dir and iwc to new a IndexWriter (writer) add documents new a Document (doc) add several fields new a Field (pathField) Field pathField = new Field("path", file.getPath(), Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS); pathField.setIndexOptions(IndexOptions.DOCS_ONLY); doc.add(pathField); new a NumericField (modifiedField) NumericField modifiedField =