kingoflolz/mesh-transformer-jax

A haiku library using the xmap/pjit operators in JAX for model parallelism of transformers. The parallelism scheme is similar to the original Megatron-LM, which is efficient on TPUs due to the high speed 2d mesh network. There is also an experimental model version which implements ZeRo style sharding.