Abstract: Agricultural heritage has the dual attributes of being a living heritage and having tourism value. Tourism development is not only conducive to improving the livelihoods of farmers in heritage areas but also provides a new path for the returning labor force. Labor mobility is related to the living inheritance of agricultural heritage and the sustainable development of heritage tourism. Research on mobility differentiation is helpful in clarifying the changing characteristics of labor migration in the tourism environment and plays an important supporting role in establishing the guidance mechanism of labor mobility. Using data obtained from a survey of villages and households, we identified the differential types according to the changes in labor flow experience and employment location, and quantitatively investigated the group characteristics and career development differences of different labor types. The following results were obtained: 1) The labor flow in heritage tourism destinations can be divided into three categories: left-behind, rural-to-urban, and return, with the first two types being the main choices of the labor force. The flow characteristics of different types of labor force are different. The migrant labor force mainly flows to Shanghai, Jiangsu, and other economically developed areas, and the migration space has a certain geographical concentration. The left-behind labor force has relatively little experience of mobility, and its production and life have typical localization attributes. The returning labor force has previously had the experience of moving away from home and has now returned to the countryside or hometown to work. The three types of labor force show obvious differences in individual characteristics, family structure, capital endowment, and geographical location. 2) Owing to differences in their mobility and employment spaces, the three types of labor force differ significantly in terms of career development. The left-behind labor force primarily engages in traditional agricultural production, with a growing trend of dual engagement as livelihoods increasingly integrate with tourism-related activities. The rural-to-urban migrant labor force is primarily employed in factories, construction, and social services. In addition to returning to agricultural production, a large proportion of the returning labor force flows to tourism management fields, such as accommodation and catering. As an important component of livelihood activities, the career development of the labor force is correlated with the family livelihood. The left-behind labor force, rural-to-urban migrant labor force, and returning labor force show significant relationships with the agriculture-oriented, non-agriculture-oriented, and tourism-oriented livelihood modes, respectively. 3) Labor flow in heritage tourism destinations is the result of multiple factors, such as individual characteristics, family structure, and capital endowment, which are important internal factors affecting the flow. Better human capital conditions are an important factor driving the labor force to enter the city, whereas advancing age drives return migration. The location condition value, economic and welfare benefits, and heritage brand value brought about by tourism development are important external factors affecting return migration. Tourism development and heritage establishment have an important impact on labor flow. However, the influence of noneconomic factors (e.g., heritage protection cognition) is not obvious, and internal and external economic factors are still the dominant factors affecting labor flow in heritage tourism destinations. To aid the orderly return of farmers and promote the returning labor force, heritage tourism sites should further strengthen their tourism development model as well as agricultural production compensation, capital capacity building, and heritage awareness cultivation to facilitate the occupational transition of returning laborers towards heritage conservation-oriented careers and promote sustainable livelihood development.