EECS/EEAP 484 Computational Intelligence, Fall 2020
Problem Set 4: Convolutional Neural Networks with Error Backpropagation
Some neural-net architectures, called convolutional networks, invoke repeated patterns of synapses. Although the network might contain many synapses, the number of independent parameters can be severely reduced using templated patterns. Convolutional networks are applicable to input patterns for which the components are not independent, but for which information has important temporal or spatial relationships among the components. Examples include strip charts, EEGs, EKGs, and (in 2-D) images.
In this problem set, you will code functions to create synapse matrices from ¡°kernels¡± (vectors of parameters) and compute sensitivities with respect to the kernel parameters. The resulting code will be applied to a (trivial) sequence-recognition task. In the specific example, there are patterns consisting of 4 integers (ranging 0 through 3). Your network should recognize the occurrence of the pattern (3,1) in the 4-digit sequence.
In the starter code, the first two layers have hard-coded synapses. The third layer has random kernel parameters. Your code should learn good kernel parameters for recognizing the specified pattern.
You will need to edit functions: compute_dE_d_kvec, compute_conv_maps, and set_weights_from_kernel. You should confirm that your dE/d_kvec values agree with the numerical estimates.
Once your network is working, try some variations. Comment on your observations regarding changes to batch size, iterations per batch, and learning-rate (eta) value.
Extend this problem set by doing one or more of the following (or some other variation you make up):
Prove that your learned kernel parameters also work for longer input sequences of values. (easy)
Extend the analysis to more input categories (e.g. digits 0 through 9). (medium difficulty) Extend the analysis to 3-digit sequences (more annoying)
Submit your code and a report, documenting success of your network on the simple application. Describe your additional experiments and your observations.