Notes and Tips

These notes include important concepts and model architectures on DNN, optimizers, hyperparameters tunning, CNN, YOLO algorithm for DeepCV, triplet loss and siamese network for one-shot learning, RNN with LSTM and GRU units, attention mechanism and Transformer architectures for NLP and CV.

Sklearn svm.SVC(kernel = “linear”) is really slow. This posts shows a trick for speeding up this process by precomputing the gram matrix of the linear kernal. This step does not require too much hussle, besides some extra matrix multiplication and re-orginizing training and testing data. But the precomputed linear kernal dramatically speed up linear SVC computation. This is another trick used in FCMA to speed up its computation process.

This post shows a simple but handy trick for efficiently speeding up pearson correlation computation. This is also the first trick used by FCMA to speed up its computation. Specifically, pearson correlation could be done use matrix multiplication after standardization (e.g., zscore)

Logistic regression and softmax are often used for modeling. This post shows how to think about these two algorithms as part of the family of generalized linear model, and what are the respective natural parameters, and how to derive their loss functions. Moreover, this post also shows that these two models can be viewed as simple neural network models with one dense layer combined with an activation function.