Life Long Learning (ntu.edu.tw)

Life Long Learning

灾难性遗忘

图片

  • transfer learning 在意在第二个任务怎么样 I can do task 2 because I have learned task 1.
  • life long learning在意第二个任务学了以后第一个怎么样 Even though I have learned task 2, I do not forget task 1.
    评估

图片

3个life long learning的解法

  1. Selective Synaptic Plasticity 突觸的 可塑性。Regularizationbased Approach 

    1. Basic Idea: Some parameters in the model are important to the previous tasks. Only change the unimportant parameters图片
      图片

    2. 一些方法文献 在ppt24页

  2. Additional Neural Resource Allocation
    1. Progressive Neural Networks 每次训练新的任务 需要额外的空间产生新的neural network。 任务量不多的时候 可以用
    2. Compacting, Picking, and Growing (CPG)
    3. PackNet
  3. Memory Reply
    1. generating data. Generating pseudo-data using generative model for previous tasks
    2. adding new classes
      Three scenarios for continual learning https://arxiv.org/abs/1904.07734

network compression

Network Compression (ntu.edu.tw)

Network Pruning

图片

  • weight pruning: the network architecture becomes irregular. hard to implement, hard to speedup 不规则 不好加速
  • Neuron pruning: The network architecture is regular. Easy to implement, easy to speedup …… 
  • why pruning: larger network is easier to optimize
    • lottery ticket hypothesis

Knowledge Distillation

  • temperature for softmax
  • teacher net to student net. Providing the information that “1” is similar to “7”

 

Parameter Quantization

  1. Using less bits to represent a value

  2. Weight clustering 

  3. Represent frequent clusters by less bits, represent rare clusters by more bits

• e.g. Huffman encoding 

binary weights: weights are always +1 or -1

Architecture Design

  1. Depthwise Convolution 
  2. Pointwise Convolution (1x1 filter)

Dynamic Computation

The network adjusts the computation it need.

Different devices, high/low battery

图片