My algorithm (Score 1)

FWIW, this algorithm was used, with some good success on a series of ASICs back in the 1980s. I have no reason to doubt that the underlying ideas are still sound.

Sometimes, the cost of the ASIC is dominated by the size of the package, other times it is the cost of the gates. I suspect that unless you have extreme I/O needs, this will turn out to be an irrelevance given modern densities; nevertheless:

• First, count the gates.
  1. Estimate the number of bit of register state and sequencer state you need. IIRC, a 1 bit register in CMOS was equivalent to 4 2-input NAND gates.
  2. Multiply the number of bits of register and sequencer state by a factor to account for the combinatorial logic directly associated with the state bits (be they combinatorial next-state logic for a state machine or datapath control logic feeding a register bank).
  3. Estimate the number of gates of combinatorial logic you need for one of your datapath bits. Multiply by the width. Repeat for all datapaths.
  4. Add any bulk storage (e.g. a dense FIFO will have little control logic per entry).
• Second count the I/O pins you need.

Choose either the gate count based answer o the pin count based answer. For the ASICs I worked on, this turned out to be amazingly accurate (within 10 or 15%) though naturally YMMV.

P.S. No warranty, express or implied, is offered with this answer!