Read any docs on the security of the product; often they'll name specific algorithms, which will narrow your search by a lot.
Hash functions
There are a few standardized hashes that are used all the time. MD5 and SHA are two of them. A hash function takes a block of data and mixes it into some state registers. The final state of the registers is output as the hash. The initial state, though, is always the same. You can search the exe for these constants to find the hashing code.
Hashing code is also full of small nonlinear functions; lots of and's, or's, and xor's, very densely packed. Hashing code doesn't have jumps--it's pages and pages of bit operations.
A hash will nearly always have three calls associated with it: the init function, the update function, and the finalize function. Finalize pads the end of the data out to a certain size and appends the length in bits to form the last block. This prevents certain attacks.
Stream ciphers
Encrypting a file is often done with a stream cipher like RC4. RC4 has an initialization loop where it fills a 256-byte block with the values 0-255.
Also, stream ciphers are going to XOR the plaintext with the random output bytes. I found the encryption code for WordPro by dumping the exe in WDASM. I wrote a small program to search for XORs where the first and second parameters weren't the same. I only had to glance at the code surrounding about 20 of them before I found it.
Block ciphers
There are also a few popular block ciphers that get used a lot. DES used to be popular; Blowfish, IDEA, are some others, and now that AES is in place, Rijndael will become much more popular. In order to make these run fast, there are optimizations that usually occur as precomputed tables (especially Blowfish). You can search on these popular implementations.
Block ciphers generally work on 64-bit or 128-bit chunks of data in a big loop.
Public Key
Public Key crypto is only done well with bignum code. Operations nearly always include modular exponentiation. Modular exponentiation is done in a loop over the bits of the exponent, where the number is squared, conditionally multiplied, and then reduced.
Good luck, and happy cracking!
--mike
grits to all the peudonyms at fraviamb