Control de sistemas usando aprendizaje de máquina

Systems control using machine learning

Jesús Martín Miguel Martínez (2023, [Tesis de maestría])

El aprendizaje por refuerzo es un paradigma del aprendizaje de máquina con un amplio desarrollo y una creciente demanda en aplicaciones que involucran toma de decisiones y control. Es un paradigma que permite el diseño de controladores que no dependen directamente del modelo que describe la dinámica del sistema. Esto es importante ya que en aplicaciones reales es frecuente que no se disponga de dichos modelos de manera precisa. Esta tesis tiene como objetivo implementar un controlador óptimo en tiempo discreto libre de modelo. La metodología elegida se basa en algoritmos de aprendizaje por refuerzo, enfocados en sistemas con espacios de estado y acción continuos a través de modelos discretos. Se utiliza el concepto de función de valor (Q-función y función V ) y la ecuación de Bellman para resolver el problema del regulador cuadrático lineal para un sistema mecánico masa-resorte-amortiguador, en casos donde se tiene conocimiento parcial y desconocimiento total del modelo. Para ambos casos las funciones de valor son definidas explícitamente por la estructura de un aproximador paramétrico, donde el vector de pesos del aproximador es sintonizado a través de un proceso iterativo de estimación de parámetros. Cuando se tiene conocimiento parcial de la dinámica se usa el método de aprendizaje por diferencias temporales en un entrenamiento episódico, que utiliza el esquema de mínimos cuadrados con mínimos cuadrados recursivos en la sintonización del crítico y descenso del gradiente en la sintonización del actor, el mejor resultado para este esquema es usando el algoritmo de iteración de valor para la solución de la ecuación de Bellman, con un resultado significativo en términos de precisión en comparación a los valores óptimos (función DLQR). Cuando se tiene desconocimiento de la dinámica se usa el algoritmo Q-learning en entrenamiento continuo, con el esquema de mínimos cuadrados con mínimos cuadrados recursivos y el esquema de mínimos cuadrados con descenso del gradiente. Ambos esquemas usan el algoritmo de iteración de política para la solución de la ecuación de Bellman, y se obtienen resultados de aproximadamente 0.001 en la medición del error cuadrático medio. Se realiza una prueba de adaptabilidad considerando variaciones que puedan suceder en los parámetros de la planta, siendo el esquema de mínimos cuadrados con mínimos cuadrados recursivos el que tiene los mejores resultados, reduciendo significativamente ...

Reinforcement learning is a machine learning paradigm with extensive development and growing demand in decision-making and control applications. This technique allows the design of controllers that do not directly depend on the model describing the system dynamics. It is useful in real-world applications, where accurate models are often unavailable. The objective of this work is to implement a modelfree discrete-time optimal controller. Through discrete models, we implemented reinforcement learning algorithms focused on systems with continuous state and action spaces. The concepts of value-function, Q-function, V -function, and the Bellman equation are employed to solve the linear quadratic regulator problem for a mass-spring-damper system in a partially known and utterly unknown model. For both cases, the value functions are explicitly defined by a parametric approximator’s structure, where the weight vector is tuned through an iterative parameter estimation process. When partial knowledge of the dynamics is available, the temporal difference learning method is used under episodic training, utilizing the least squares with a recursive least squares scheme for tuning the critic and gradient descent for the actor´s tuning. The best result for this scheme is achieved using the value iteration algorithm for solving the Bellman equation, yielding significant improvements in approximating the optimal values (DLQR function). When the dynamics are entirely unknown, the Q-learning algorithm is employed in continuous training, employing the least squares with recursive least squares and the gradient descent schemes. Both schemes use the policy iteration algorithm to solve the Bellman equation, and the system’s response using the obtained values was compared to the one using the theoretical optimal values, yielding approximately zero mean squared error between them. An adaptability test is conducted considering variations that may occur in plant parameters, with the least squares with recursive least squares scheme yielding the best results, significantly reducing the number of iterations required for convergence to optimal values.

aprendizaje por refuerzo, control óptimo, control adaptativo, sistemas mecánicos, libre de modelo, dinámica totalmente desconocida, aproximación paramétrica, Q-learning, iteración de política reinforcement learning, optimal control, adaptive control, mechanical systems, modelfree, utterly unknown dynamics, parametric approximation, Q-learning, policy iteration INGENIERÍA Y TECNOLOGÍA CIENCIAS TECNOLÓGICAS TECNOLOGÍA DE LOS ORDENADORES INTELIGENCIA ARTIFICIAL INTELIGENCIA ARTIFICIAL

Rapid effects of marine reserves via larval dispersal

Richard Cudney Bueno (2009, [Artículo])

Marine reserves have been advocated worldwide as conservation and fishery management tools. It is argued that they can protect ecosystems and also benefit fisheries via density-dependent spillover of adults and enhanced larval dispersal into fishing areas. However, while evidence has shown that marine reserves can meet conservation targets, their effects on fisheries are less understood. In particular, the basic question of if and over what temporal and spatial scales reserves can benefit fished populations via larval dispersal remains unanswered. We tested predictions of a larval transport model for a marine reserve network in the Gulf of California, Mexico, via field oceanography and repeated density counts of recently settled juvenile commercial mollusks before and after reserve establishment. We show that local retention of larvae within a reserve network can take place with enhanced, but spatially-explicit, recruitment to local fisheries. Enhancement occurred rapidly (2 yrs), with up to a three-fold increase in density of juveniles found in fished areas at the downstream edge of the reserve network, but other fishing areas within the network were unaffected. These findings were consistent with our model predictions. Our findings underscore the potential benefits of protecting larval sources and show that enhancement in recruitment can be manifested rapidly. However, benefits can be markedly variable within a local seascape. Hence, effects of marine reserve networks, positive or negative, may be overlooked when only focusing on overall responses and not considering finer spatially-explicit responses within a reserve network and its adjacent fishing grounds. Our results therefore call for future research on marine reserves that addresses this variability in order to help frame appropriate scenarios for the spatial management scales of interest. © 2009 Cudney-Bueno et al.

article, environmental monitoring, fishery, larva, marine environment, marine species, Mexico, mollusc, nonhuman, oceanography, prediction, animal, biology, environmental protection, food industry, geography, growth, development and aging, larva, met CIENCIAS FÍSICO MATEMÁTICAS Y CIENCIAS DE LA TIERRA CIENCIAS DE LA TIERRA Y DEL ESPACIO OCEANOGRAFÍA OCEANOGRAFÍA

High quality diet improves lipid metabolic profile and breeding performance in the blue-footed booby, a long-lived seabird

ERICK GONZALEZ MEDINA (2018, [Artículo])

Understanding the role of diet in the physiological condition of adults during reproduction and hence its effect on reproductive performance is fundamental to understand reproductive strategies in long-lived animals. In birds, little is known about the influence of the quality of food consumed at the beginning of the reproductive period and its short-term effects on reproductive performance. To assess the role of diet in the physiological condition of female blue-footed booby, Sula nebouxii (BFBO), during reproduction we evaluated whether individual differences in diet (assessed by using δ13C and δ15N values of whole blood from female birds and muscle tissue of the principal prey species) prior to egg laying and during incubation influenced their lipid metabolic profile (measured as triglyceride levels and C:N ratio) and their reproductive performance (defined by laying date, clutch size and hatching success). Females with higher δ15N values in their blood during the courtship and incubation periods had a higher lipid metabolic profile, earlier laying date, greater clutch size (2–3 eggs) and higher hatching success. Females that laid earlier and more eggs (2–3 eggs) consumed more Pacific anchoveta (Cetengraulis mysticetus) and Pacific thread herring (Opisthonema libertate) than did other females. These two prey species also had high amounts of lipids (C:N ratio) and caloric content (Kcal/g fresh weight). The quality of food consumed by females at the beginning of reproduction affected their physiological condition, as well as their short-term reproductive performance. Our work emphasizes the importance of determining the influence of food quality during reproduction to understand the reproductive decisions and consequences in long-lived animals. © 2018 González-Medina et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

nitrogen 15, triacylglycerol, lipid, animal experiment, Article, breeding, carbon nitrogen ratio, clutch size, controlled study, courtship, diet, egg laying, female, food intake, hatching, lipid metabolism, muscle tissue, nonhuman, prey, reproduction CIENCIAS FÍSICO MATEMÁTICAS Y CIENCIAS DE LA TIERRA CIENCIAS DE LA TIERRA Y DEL ESPACIO OCEANOGRAFÍA OCEANOGRAFÍA

The Pacific harbor seal gut microbiota in Mexico: Its relationship with diet and functional inferences

ARLETTE MARIMAR PACHECO SANDOVAL (2019, [Artículo])

Diet is a primary driver of the composition of gut microbiota and is considered one of the main routes of microbial colonization. Prey identification is fundamental for correlating the diet with the presence of particular microbial groups. The present study examined how diet influenced the composition and function of the gut microbiota of the Pacific harbor seal (Phoca vitulina richardii) in order to better understand the role of prey consumption in shaping its microbiota. This species is a good indicator of the quality of the local environment due to both its foraging and haul-out site fidelity. DNA was extracted from 20 fecal samples collected from five harbor seal colonies located in Baja California, Mexico. The V4 region of 16S rRNA gene was amplified and sequenced using the Illumina technology. Results showed that the gut microbiota of the harbor seals was dominated by the phyla Firmicutes (37%), Bacteroidetes (26%) and Fusobacteria (26%) and revealed significant differences in its composition among the colonies. Funtional analysis using the PICRUSt software suggests a high number of pathways involved in the basal metabolism, such as those for carbohydrates (22%) and amino acids (20%), and those related to the degradation of persistent environmental pollutants. In addition, a DNA metabarcoding analysis of the same samples, via the amplification and sequencing of the mtRNA 16S and rRNA 18S genes, was used to identify the prey consumed by harbor seals revealing the consumption of prey with mainly demersal habits. Functional redundancy in the seal gut microbiota was observed, irrespective of diet or location. Our results indicate that the frequency of occurrence of specific prey in the harbor seal diet plays an important role in shaping the composition of the gut microbiota of harbor seals by influencing the relative abundance of specific groups of gut microorganisms. A significant relationship was found among diet, gut microbiota composition and OTUs assigned to a particular metabolic pathway. © 2019 Pacheco-Sandoval et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

RNA 16S, RNA 18S, amino acid analysis, animal food, Article, bacterium colony, Bacteroidetes, basal metabolic rate, biodegradation, controlled study, DNA barcoding, feces analysis, Firmicutes, Fusobacteria, intestine flora, metabolism, Mexico, microb BIOLOGÍA Y QUÍMICA CIENCIAS DE LA VIDA BIOLOGÍA ANIMAL (ZOOLOGÍA) BIOLOGÍA ANIMAL (ZOOLOGÍA)