In this article, we will explore the recent development of physical literacy in the Hong Kong context and how the concept and operation of physical literacy implicitly exist at different levels of the Hong Kong education system. The Physical Education profession will be introduced. The development of physical literacy in terms of research and operationalization in primary, secondary, and tertiary education will then be discussed. We will go on to explore the challenges of extending the impact of physical literacy to the field of public health in Hong Kong. The article will end with a closing remark adopting the Chinese philosophies of Confucianism and Taoism to justify the belief that physical literacy is both implicitly and invisibly rooted in the Hong Kong Chinese culture. Copyright ©MDPI AG.